2024-04-25T06:51:01.918 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T06:51:01.918 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672918 --owner scheduled_nmordech@teuthology --verbose --name nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi --description rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} -- /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672918/orig.config.yaml 2024-04-25T06:51:01.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T06:51:02.013 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672918 branch: main description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7672918' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f workunit: branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full sha1: a08304379f921e5e229e92c75ede8800059313ff owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7777 sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full suite_path: /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa suite_relpath: qa suite_repo: https://github.com/NitzanMordhai/ceph.git suite_sha1: a08304379f921e5e229e92c75ede8800059313ff targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl61yAxke/mMA8yxMRDx53YQC9sR2MFJJ2fs7L94qzr34WUrpm5l61SWcRI5oGfIRGeSul5eV7aDgDBpslOSqw= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_06:28:28 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T06:51:02.013 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa; will attempt to use it 2024-04-25T06:51:02.014 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa/tasks 2024-04-25T06:51:02.014 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T06:51:02.015 INFO:teuthology.task.internal:Checking packages... 2024-04-25T06:51:02.039 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f' 2024-04-25T06:51:02.039 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T06:51:02.039 INFO:teuthology.packaging:ref: None 2024-04-25T06:51:02.039 INFO:teuthology.packaging:tag: None 2024-04-25T06:51:02.039 INFO:teuthology.packaging:branch: main 2024-04-25T06:51:02.039 INFO:teuthology.packaging:sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T06:51:02.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-04-25T06:51:02.276 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3253-g4e213a58-1jammy 2024-04-25T06:51:02.278 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T06:51:02.285 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T06:51:02.285 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T06:51:02.295 INFO:teuthology.task.internal:Saving configuration 2024-04-25T06:51:02.304 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T06:51:02.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T06:51:02.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672918', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 06:45:44.065375', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK3+ydXK7T/jDf7orx+SNJYC3GVus9QXaocGo/FK5wEw'} 2024-04-25T06:51:02.334 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T06:51:02.390 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-25T06:51:02.390 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T06:51:02.430 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcb108de3a0>, signals=[15]) 2024-04-25T06:51:02.430 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T06:51:02.439 INFO:teuthology.task.internal:Opening connections... 2024-04-25T06:51:02.439 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T06:51:02.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:51:02.523 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T06:51:02.530 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-04-25T06:51:02.639 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-04-25T06:51:02.639 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Ubuntu" 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="22.04" 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_CODENAME=jammy 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:ID=ubuntu 2024-04-25T06:51:02.687 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE=debian 2024-04-25T06:51:02.688 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T06:51:02.688 INFO:teuthology.orchestra.run.smithi031.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T06:51:02.688 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T06:51:02.688 INFO:teuthology.orchestra.run.smithi031.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T06:51:02.688 INFO:teuthology.orchestra.run.smithi031.stdout:UBUNTU_CODENAME=jammy 2024-04-25T06:51:02.688 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-04-25T06:51:02.711 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T06:51:02.721 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T06:51:02.729 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T06:51:02.729 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-04-25T06:51:02.734 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T06:51:02.743 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T06:51:02.743 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-04-25T06:51:02.789 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T06:51:02.821 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T06:51:02.837 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T06:51:02.837 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T06:51:02.838 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T06:51:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:51:02.843 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-25T06:51:02.891 INFO:teuthology.orchestra.run.smithi031.stdout:5.15.0-56-generic 2024-04-25T06:51:02.892 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.15.0-56-generic 2024-04-25T06:51:02.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get clean 2024-04-25T06:51:03.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2024-04-25T06:51:03.158 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:51:03.159 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T06:51:03.159 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T06:51:03.248 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T06:51:03.417 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T06:51:03.497 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T06:51:03.519 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T06:51:03.563 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T06:51:03.566 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T06:51:03.566 INFO:teuthology.orchestra.run.smithi031.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T06:51:03.590 INFO:teuthology.orchestra.run.smithi031.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T06:51:03.591 INFO:teuthology.orchestra.run.smithi031.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T06:51:03.594 INFO:teuthology.orchestra.run.smithi031.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:51:03.594 INFO:teuthology.orchestra.run.smithi031.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T06:51:03.599 INFO:teuthology.orchestra.run.smithi031.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T06:51:03.599 INFO:teuthology.orchestra.run.smithi031.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T06:51:03.612 INFO:teuthology.orchestra.run.smithi031.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T06:51:03.613 INFO:teuthology.orchestra.run.smithi031.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T06:51:03.614 INFO:teuthology.orchestra.run.smithi031.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T06:51:03.637 INFO:teuthology.orchestra.run.smithi031.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T06:51:03.637 INFO:teuthology.orchestra.run.smithi031.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T06:51:03.637 INFO:teuthology.orchestra.run.smithi031.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T06:51:03.638 INFO:teuthology.orchestra.run.smithi031.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T06:51:03.638 INFO:teuthology.orchestra.run.smithi031.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T06:51:03.643 INFO:teuthology.orchestra.run.smithi031.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T06:51:03.643 INFO:teuthology.orchestra.run.smithi031.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T06:51:03.654 INFO:teuthology.orchestra.run.smithi031.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T06:51:03.657 INFO:teuthology.orchestra.run.smithi031.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:51:03.657 INFO:teuthology.orchestra.run.smithi031.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T06:51:03.666 INFO:teuthology.orchestra.run.smithi031.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T06:51:03.672 INFO:teuthology.orchestra.run.smithi031.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T06:51:03.681 INFO:teuthology.orchestra.run.smithi031.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T06:51:03.681 INFO:teuthology.orchestra.run.smithi031.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T06:51:03.684 INFO:teuthology.orchestra.run.smithi031.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T06:51:03.685 INFO:teuthology.orchestra.run.smithi031.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T06:51:03.685 INFO:teuthology.orchestra.run.smithi031.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T06:51:03.685 INFO:teuthology.orchestra.run.smithi031.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-25T06:51:03.686 INFO:teuthology.orchestra.run.smithi031.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-25T06:51:03.715 INFO:teuthology.orchestra.run.smithi031.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T06:51:03.715 INFO:teuthology.orchestra.run.smithi031.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T06:51:03.715 INFO:teuthology.orchestra.run.smithi031.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-25T06:51:03.716 INFO:teuthology.orchestra.run.smithi031.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-25T06:51:03.716 INFO:teuthology.orchestra.run.smithi031.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T06:51:03.735 INFO:teuthology.orchestra.run.smithi031.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T06:51:09.547 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 13.3 MB in 2s (6,846 kB/s) 2024-04-25T06:51:11.093 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-04-25T06:51:11.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:51:11.175 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-04-25T06:51:11.382 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-04-25T06:51:11.382 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-04-25T06:51:11.564 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:51:11.564 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:51:11.564 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:51:11.565 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:51:11.565 INFO:teuthology.orchestra.run.smithi031.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T06:51:11.565 INFO:teuthology.orchestra.run.smithi031.stdout: python2.7-minimal 2024-04-25T06:51:11.565 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:51:11.566 INFO:teuthology.orchestra.run.smithi031.stdout:The following additional packages will be installed: 2024-04-25T06:51:11.566 INFO:teuthology.orchestra.run.smithi031.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:51:11.566 INFO:teuthology.orchestra.run.smithi031.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T06:51:11.567 INFO:teuthology.orchestra.run.smithi031.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T06:51:11.568 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2024-04-25T06:51:11.568 INFO:teuthology.orchestra.run.smithi031.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T06:51:11.568 INFO:teuthology.orchestra.run.smithi031.stdout:Recommended packages: 2024-04-25T06:51:11.568 INFO:teuthology.orchestra.run.smithi031.stdout: thermald 2024-04-25T06:51:11.628 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2024-04-25T06:51:11.628 INFO:teuthology.orchestra.run.smithi031.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:51:11.628 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T06:51:11.629 INFO:teuthology.orchestra.run.smithi031.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T06:51:11.630 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages will be upgraded: 2024-04-25T06:51:11.630 INFO:teuthology.orchestra.run.smithi031.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T06:51:11.689 INFO:teuthology.orchestra.run.smithi031.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:51:11.689 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 113 MB of archives. 2024-04-25T06:51:11.689 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T06:51:11.689 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T06:51:12.249 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T06:51:12.393 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T06:51:13.784 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T06:51:13.784 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T06:51:13.785 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T06:51:14.084 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T06:51:14.134 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T06:51:14.690 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 113 MB in 2s (45.4 MB/s) 2024-04-25T06:51:14.871 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T06:51:16.959 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T06:51:16.963 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:51:17.242 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:19.284 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T06:51:19.306 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:51:19.400 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:19.871 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T06:51:19.891 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:51:19.929 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:26.511 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:51:26.636 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:51:26.952 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:51:27.073 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:51:27.359 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T06:51:27.382 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T06:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:51:32.859 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T06:51:32.896 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:51:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:35.679 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:51:35.803 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:51:36.104 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:51:36.258 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:36.384 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T06:51:36.513 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:37.885 INFO:teuthology.orchestra.run.smithi031.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:51:37.885 INFO:teuthology.orchestra.run.smithi031.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T06:51:37.885 INFO:teuthology.orchestra.run.smithi031.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:51:37.885 INFO:teuthology.orchestra.run.smithi031.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T06:51:38.048 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:39.834 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:41.243 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T06:51:41.385 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T06:51:41.512 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:51:41.582 INFO:teuthology.orchestra.run.smithi031.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T06:51:41.582 INFO:teuthology.orchestra.run.smithi031.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:51:56.107 INFO:teuthology.orchestra.run.smithi031.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T06:51:56.107 INFO:teuthology.orchestra.run.smithi031.stdout:Sourcing file `/etc/default/grub' 2024-04-25T06:51:56.120 INFO:teuthology.orchestra.run.smithi031.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:51:56.139 INFO:teuthology.orchestra.run.smithi031.stdout:Generating grub configuration file ... 2024-04-25T06:51:56.393 INFO:teuthology.orchestra.run.smithi031.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:51:56.410 INFO:teuthology.orchestra.run.smithi031.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:51:56.723 INFO:teuthology.orchestra.run.smithi031.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:51:56.729 INFO:teuthology.orchestra.run.smithi031.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:51:56.799 INFO:teuthology.orchestra.run.smithi031.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:51:56.805 INFO:teuthology.orchestra.run.smithi031.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:51:56.964 INFO:teuthology.orchestra.run.smithi031.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:51:56.964 INFO:teuthology.orchestra.run.smithi031.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:51:56.964 INFO:teuthology.orchestra.run.smithi031.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:51:57.008 INFO:teuthology.orchestra.run.smithi031.stdout:done 2024-04-25T06:51:57.506 DEBUG:teuthology.orchestra.run.smithi031:> dpkg -s linux-image-generic 2024-04-25T06:51:57.530 INFO:teuthology.orchestra.run.smithi031.stdout:Package: linux-image-generic 2024-04-25T06:51:57.530 INFO:teuthology.orchestra.run.smithi031.stdout:Status: install ok installed 2024-04-25T06:51:57.530 INFO:teuthology.orchestra.run.smithi031.stdout:Priority: optional 2024-04-25T06:51:57.530 INFO:teuthology.orchestra.run.smithi031.stdout:Section: kernel 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Installed-Size: 21 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: amd64 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Source: linux-meta 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Version: 5.15.0.105.102 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Recommends: thermald 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:Description: Generic Linux kernel image 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:51:57.531 INFO:teuthology.orchestra.run.smithi031.stdout: available. 2024-04-25T06:51:57.531 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T06:51:57.532 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T06:51:57.532 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T06:51:57.532 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T06:51:57.532 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T06:51:57.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get clean 2024-04-25T06:51:57.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2024-04-25T06:51:57.876 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T06:51:58.001 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:51:58.074 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T06:51:58.156 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T06:51:59.308 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-04-25T06:51:59.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:51:59.387 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-04-25T06:51:59.590 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-04-25T06:51:59.591 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-04-25T06:51:59.782 INFO:teuthology.orchestra.run.smithi031.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T06:51:59.783 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:51:59.783 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:51:59.783 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:51:59.783 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:51:59.783 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T06:51:59.783 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T06:51:59.784 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:51:59.826 INFO:teuthology.orchestra.run.smithi031.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:51:59.828 DEBUG:teuthology.orchestra.run.smithi031:> dpkg -s linux-image-generic 2024-04-25T06:51:59.846 INFO:teuthology.orchestra.run.smithi031.stdout:Package: linux-image-generic 2024-04-25T06:51:59.846 INFO:teuthology.orchestra.run.smithi031.stdout:Status: install ok installed 2024-04-25T06:51:59.846 INFO:teuthology.orchestra.run.smithi031.stdout:Priority: optional 2024-04-25T06:51:59.846 INFO:teuthology.orchestra.run.smithi031.stdout:Section: kernel 2024-04-25T06:51:59.846 INFO:teuthology.orchestra.run.smithi031.stdout:Installed-Size: 21 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: amd64 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Source: linux-meta 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Version: 5.15.0.105.102 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:51:59.847 INFO:teuthology.orchestra.run.smithi031.stdout:Recommends: thermald 2024-04-25T06:51:59.848 INFO:teuthology.orchestra.run.smithi031.stdout:Description: Generic Linux kernel image 2024-04-25T06:51:59.848 INFO:teuthology.orchestra.run.smithi031.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:51:59.848 INFO:teuthology.orchestra.run.smithi031.stdout: available. 2024-04-25T06:51:59.848 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-04-25T06:51:59.891 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.NwqM3SCg4d 2024-04-25T06:51:59.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NwqM3SCg4d 2024-04-25T06:51:59.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.NwqM3SCg4d 2024-04-25T06:52:00.061 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.NwqM3SCg4d is 10KB 2024-04-25T06:52:00.074 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.NwqM3SCg4d 2024-04-25T06:52:00.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T06:52:00.134 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T06:52:00.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T06:52:00.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T06:52:00.202 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T06:52:00.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo update-grub 2024-04-25T06:52:01.409 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub' 2024-04-25T06:52:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:52:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-04-25T06:52:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:52:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:52:01.786 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:52:01.792 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:52:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:52:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:52:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:52:01.958 INFO:teuthology.orchestra.run.smithi031.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:52:01.958 INFO:teuthology.orchestra.run.smithi031.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:52:01.990 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-04-25T06:52:01.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-04-25T06:52:32.022 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T06:52:32.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T06:52:32.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:53:32.070 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T06:53:41.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T06:53:41.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:53:41.281 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-25T06:53:41.811 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T06:53:41.811 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T06:53:41.811 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-25T06:53:41.857 INFO:teuthology.orchestra.run.smithi031.stdout:5.15.0-105-generic 2024-04-25T06:53:41.857 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T06:53:41.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T06:53:41.858 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T06:53:42.858 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T06:53:42.859 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T06:53:42.917 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-04-25T06:53:42.947 DEBUG:teuthology.parallel:result is None 2024-04-25T06:53:42.948 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T06:53:42.955 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T06:53:42.955 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T06:53:42.961 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T06:53:42.967 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T06:53:42.975 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T06:53:42.975 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T06:53:43.026 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T06:53:43.033 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T06:53:43.033 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T06:53:43.079 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:53:43.087 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:53:43.089 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T06:53:43.095 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T06:53:43.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T06:53:43.146 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T06:53:43.154 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T06:53:43.154 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T06:53:43.194 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T06:53:43.242 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T06:53:43.290 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T06:53:43.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T06:53:43.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-04-25T06:53:43.422 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T06:53:43.430 INFO:teuthology.task.internal:Starting timer... 2024-04-25T06:53:43.430 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T06:53:43.445 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T06:53:43.453 DEBUG:teuthology.task.selinux:Excluding smithi031: OS 'ubuntu' does not support SELinux 2024-04-25T06:53:43.454 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T06:53:43.454 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T06:53:43.454 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T06:53:43.454 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T06:53:43.463 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T06:53:43.469 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T06:53:43.584 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T06:53:43.598 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T06:53:43.600 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T07:00:47.417 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2024-04-25T07:00:47.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T07:00:47.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T07:00:47.507 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-25T07:00:47.586 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T07:00:47.587 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T07:00:47.597 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T07:00:47.598 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T07:00:47.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T07:00:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T07:00:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Command line: ntpd -gq 2024-04-25T07:00:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: ---------------------------------------------------- 2024-04-25T07:00:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T07:00:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T07:00:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: corporation. Support and training for ntp-4 are 2024-04-25T07:00:47.659 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: available at https://www.nwtime.org/support 2024-04-25T07:00:47.659 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: ---------------------------------------------------- 2024-04-25T07:00:47.659 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: proto: precision = 0.066 usec (-24) 2024-04-25T07:00:47.659 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: basedate set to 2022-02-04 2024-04-25T07:00:47.659 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: gps base set to 2022-02-06 (week 2196) 2024-04-25T07:00:47.660 INFO:teuthology.orchestra.run.smithi031.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T07:00:47.660 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T07:00:47.660 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T07:00:47.661 INFO:teuthology.orchestra.run.smithi031.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T07:00:47.661 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T07:00:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T07:00:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T07:00:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listen normally on 3 ens1f0 172.21.15.31:123 2024-04-25T07:00:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listen normally on 4 lo [::1]:123 2024-04-25T07:00:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:caba%4]:123 2024-04-25T07:00:47.662 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:47 ntpd[17105]: Listening on routing socket on fd #22 for interface updates 2024-04-25T07:00:48.658 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:48 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:48 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:48 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:48 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:48 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:48.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:48 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:49 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:49 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:49 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:49 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:49 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:49.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:49 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:50.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:50 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:51 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:51 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:51 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:51 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:51 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:51 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:52.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:52.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:52.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:52.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:52 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:53.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:53 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:53.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:53 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:53.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:53 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:53.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:53 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:53.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:53 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:53.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:53 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:54 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stdout:25 Apr 07:00:55 ntpd[17105]: ntpd: time slew -0.000402 s 2024-04-25T07:00:55.659 INFO:teuthology.orchestra.run.smithi031.stdout:ntpd: time slew -0.000402s 2024-04-25T07:00:55.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T07:00:55.660 INFO:teuthology.orchestra.run.smithi031.stderr:25 Apr 07:00:55 ntpd[17105]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T07:00:55.714 INFO:teuthology.orchestra.run.smithi031.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T07:00:55.714 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================== 2024-04-25T07:00:55.714 INFO:teuthology.orchestra.run.smithi031.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:00:55.714 INFO:teuthology.orchestra.run.smithi031.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:00:55.714 INFO:teuthology.orchestra.run.smithi031.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:00:55.714 INFO:teuthology.orchestra.run.smithi031.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:00:55.715 INFO:teuthology.run_tasks:Running task install... 2024-04-25T07:00:55.726 DEBUG:teuthology.task.install:project ceph 2024-04-25T07:00:55.726 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'}} 2024-04-25T07:00:55.726 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'} 2024-04-25T07:00:55.726 INFO:teuthology.task.install:Using flavor: default 2024-04-25T07:00:55.732 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T07:00:55.732 INFO:teuthology.task.install:extra packages: [] 2024-04-25T07:00:55.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-key list | grep Ceph 2024-04-25T07:00:55.818 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T07:00:55.845 INFO:teuthology.orchestra.run.smithi031.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T07:00:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T07:00:55.846 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-25T07:00:55.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T07:00:55.992 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default/ 2024-04-25T07:00:55.992 INFO:teuthology.task.install.deb:Package version is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:00:56.232 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T07:00:56.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T07:00:56.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2024-04-25T07:00:56.425 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T07:00:56.425 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T07:00:56.438 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T07:00:56.459 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T07:00:56.474 INFO:teuthology.orchestra.run.smithi031.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T07:00:56.496 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T07:00:56.518 INFO:teuthology.orchestra.run.smithi031.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T07:00:56.539 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-04-25T07:00:56.567 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-04-25T07:00:56.801 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 29.3 kB in 0s (78.5 kB/s) 2024-04-25T07:00:57.761 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-04-25T07:00:57.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3227-gc37f63b3-1jammy cephadm=19.0.0-3227-gc37f63b3-1jammy ceph-mds=19.0.0-3227-gc37f63b3-1jammy ceph-mgr=19.0.0-3227-gc37f63b3-1jammy ceph-common=19.0.0-3227-gc37f63b3-1jammy ceph-fuse=19.0.0-3227-gc37f63b3-1jammy ceph-test=19.0.0-3227-gc37f63b3-1jammy ceph-volume=19.0.0-3227-gc37f63b3-1jammy radosgw=19.0.0-3227-gc37f63b3-1jammy python3-rados=19.0.0-3227-gc37f63b3-1jammy python3-rgw=19.0.0-3227-gc37f63b3-1jammy python3-cephfs=19.0.0-3227-gc37f63b3-1jammy python3-rbd=19.0.0-3227-gc37f63b3-1jammy libcephfs2=19.0.0-3227-gc37f63b3-1jammy libcephfs-dev=19.0.0-3227-gc37f63b3-1jammy librados2=19.0.0-3227-gc37f63b3-1jammy librbd1=19.0.0-3227-gc37f63b3-1jammy rbd-fuse=19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:00:57.831 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-04-25T07:00:58.021 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-04-25T07:00:58.021 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-04-25T07:00:58.210 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T07:00:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T07:00:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T07:00:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T07:00:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T07:00:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T07:00:58.211 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T07:00:58.212 INFO:teuthology.orchestra.run.smithi031.stdout:The following additional packages will be installed: 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T07:00:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-25T07:00:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet zip 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout:Recommended packages: 2024-04-25T07:00:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: btrfs-tools 2024-04-25T07:00:58.317 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2024-04-25T07:00:58.318 INFO:teuthology.orchestra.run.smithi031.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T07:00:58.318 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T07:00:58.318 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-25T07:00:58.318 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-25T07:00:58.318 INFO:teuthology.orchestra.run.smithi031.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-25T07:00:58.318 INFO:teuthology.orchestra.run.smithi031.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-25T07:00:58.319 INFO:teuthology.orchestra.run.smithi031.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-25T07:00:58.319 INFO:teuthology.orchestra.run.smithi031.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-25T07:00:58.319 INFO:teuthology.orchestra.run.smithi031.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-25T07:00:58.319 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-25T07:00:58.320 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-25T07:00:58.321 INFO:teuthology.orchestra.run.smithi031.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-25T07:00:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T07:00:58.388 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 181 MB of archives. 2024-04-25T07:00:58.388 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-25T07:00:58.388 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T07:00:58.489 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3227-gc37f63b3-1jammy [3,556 kB] 2024-04-25T07:00:58.495 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T07:00:58.499 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T07:00:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T07:00:58.597 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T07:00:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T07:00:58.621 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T07:00:58.627 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T07:00:58.628 INFO:teuthology.orchestra.run.smithi031.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T07:00:58.629 INFO:teuthology.orchestra.run.smithi031.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T07:00:58.640 INFO:teuthology.orchestra.run.smithi031.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T07:00:58.643 INFO:teuthology.orchestra.run.smithi031.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T07:00:58.646 INFO:teuthology.orchestra.run.smithi031.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T07:00:58.647 INFO:teuthology.orchestra.run.smithi031.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T07:00:58.648 INFO:teuthology.orchestra.run.smithi031.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T07:00:58.661 INFO:teuthology.orchestra.run.smithi031.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T07:00:58.664 INFO:teuthology.orchestra.run.smithi031.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T07:00:58.666 INFO:teuthology.orchestra.run.smithi031.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T07:00:58.667 INFO:teuthology.orchestra.run.smithi031.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T07:00:58.682 INFO:teuthology.orchestra.run.smithi031.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T07:00:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T07:00:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T07:00:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T07:00:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T07:00:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T07:00:58.703 INFO:teuthology.orchestra.run.smithi031.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T07:00:58.703 INFO:teuthology.orchestra.run.smithi031.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T07:00:58.707 INFO:teuthology.orchestra.run.smithi031.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T07:00:58.707 INFO:teuthology.orchestra.run.smithi031.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T07:00:58.724 INFO:teuthology.orchestra.run.smithi031.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T07:00:58.725 INFO:teuthology.orchestra.run.smithi031.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T07:00:58.726 INFO:teuthology.orchestra.run.smithi031.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T07:00:58.726 INFO:teuthology.orchestra.run.smithi031.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T07:00:58.727 INFO:teuthology.orchestra.run.smithi031.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T07:00:58.728 INFO:teuthology.orchestra.run.smithi031.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T07:00:58.745 INFO:teuthology.orchestra.run.smithi031.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T07:00:58.745 INFO:teuthology.orchestra.run.smithi031.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T07:00:58.752 INFO:teuthology.orchestra.run.smithi031.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T07:00:58.752 INFO:teuthology.orchestra.run.smithi031.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T07:00:58.765 INFO:teuthology.orchestra.run.smithi031.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T07:00:58.766 INFO:teuthology.orchestra.run.smithi031.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T07:00:58.768 INFO:teuthology.orchestra.run.smithi031.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T07:00:58.770 INFO:teuthology.orchestra.run.smithi031.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T07:00:58.771 INFO:teuthology.orchestra.run.smithi031.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T07:00:58.772 INFO:teuthology.orchestra.run.smithi031.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T07:00:58.786 INFO:teuthology.orchestra.run.smithi031.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T07:00:58.790 INFO:teuthology.orchestra.run.smithi031.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T07:00:58.790 INFO:teuthology.orchestra.run.smithi031.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T07:00:58.846 INFO:teuthology.orchestra.run.smithi031.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T07:00:58.846 INFO:teuthology.orchestra.run.smithi031.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3227-gc37f63b3-1jammy [3,224 kB] 2024-04-25T07:00:58.846 INFO:teuthology.orchestra.run.smithi031.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T07:00:58.870 INFO:teuthology.orchestra.run.smithi031.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T07:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T07:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T07:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T07:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T07:00:58.893 INFO:teuthology.orchestra.run.smithi031.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T07:00:58.895 INFO:teuthology.orchestra.run.smithi031.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T07:00:58.895 INFO:teuthology.orchestra.run.smithi031.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T07:00:58.910 INFO:teuthology.orchestra.run.smithi031.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T07:00:58.910 INFO:teuthology.orchestra.run.smithi031.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T07:00:58.916 INFO:teuthology.orchestra.run.smithi031.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T07:00:58.917 INFO:teuthology.orchestra.run.smithi031.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T07:00:58.917 INFO:teuthology.orchestra.run.smithi031.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T07:00:58.918 INFO:teuthology.orchestra.run.smithi031.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T07:00:58.918 INFO:teuthology.orchestra.run.smithi031.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T07:00:58.937 INFO:teuthology.orchestra.run.smithi031.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T07:00:58.940 INFO:teuthology.orchestra.run.smithi031.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T07:00:58.942 INFO:teuthology.orchestra.run.smithi031.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T07:00:58.943 INFO:teuthology.orchestra.run.smithi031.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T07:00:58.944 INFO:teuthology.orchestra.run.smithi031.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T07:00:58.956 INFO:teuthology.orchestra.run.smithi031.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3227-gc37f63b3-1jammy [712 kB] 2024-04-25T07:00:58.957 INFO:teuthology.orchestra.run.smithi031.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T07:00:58.961 INFO:teuthology.orchestra.run.smithi031.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T07:00:58.962 INFO:teuthology.orchestra.run.smithi031.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T07:00:58.962 INFO:teuthology.orchestra.run.smithi031.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T07:00:58.967 INFO:teuthology.orchestra.run.smithi031.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T07:00:58.978 INFO:teuthology.orchestra.run.smithi031.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T07:00:58.982 INFO:teuthology.orchestra.run.smithi031.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T07:00:58.982 INFO:teuthology.orchestra.run.smithi031.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T07:00:58.982 INFO:teuthology.orchestra.run.smithi031.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T07:00:58.983 INFO:teuthology.orchestra.run.smithi031.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T07:00:58.985 INFO:teuthology.orchestra.run.smithi031.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3227-gc37f63b3-1jammy [356 kB] 2024-04-25T07:00:58.994 INFO:teuthology.orchestra.run.smithi031.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3227-gc37f63b3-1jammy [32.8 kB] 2024-04-25T07:00:58.995 INFO:teuthology.orchestra.run.smithi031.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3227-gc37f63b3-1jammy [180 kB] 2024-04-25T07:00:58.999 INFO:teuthology.orchestra.run.smithi031.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T07:00:59.000 INFO:teuthology.orchestra.run.smithi031.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T07:00:59.007 INFO:teuthology.orchestra.run.smithi031.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T07:00:59.008 INFO:teuthology.orchestra.run.smithi031.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T07:00:59.008 INFO:teuthology.orchestra.run.smithi031.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T07:00:59.011 INFO:teuthology.orchestra.run.smithi031.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3227-gc37f63b3-1jammy [67.1 kB] 2024-04-25T07:00:59.013 INFO:teuthology.orchestra.run.smithi031.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3227-gc37f63b3-1jammy [330 kB] 2024-04-25T07:00:59.020 INFO:teuthology.orchestra.run.smithi031.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T07:00:59.021 INFO:teuthology.orchestra.run.smithi031.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3227-gc37f63b3-1jammy [6,893 kB] 2024-04-25T07:00:59.021 INFO:teuthology.orchestra.run.smithi031.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T07:00:59.248 INFO:teuthology.orchestra.run.smithi031.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3227-gc37f63b3-1jammy [112 kB] 2024-04-25T07:00:59.251 INFO:teuthology.orchestra.run.smithi031.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3227-gc37f63b3-1jammy [466 kB] 2024-04-25T07:00:59.266 INFO:teuthology.orchestra.run.smithi031.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3227-gc37f63b3-1jammy [25.8 MB] 2024-04-25T07:01:00.192 INFO:teuthology.orchestra.run.smithi031.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3227-gc37f63b3-1jammy [5,092 kB] 2024-04-25T07:01:00.280 INFO:teuthology.orchestra.run.smithi031.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3227-gc37f63b3-1jammy [244 kB] 2024-04-25T07:01:00.287 INFO:teuthology.orchestra.run.smithi031.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T07:01:00.292 INFO:teuthology.orchestra.run.smithi031.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3227-gc37f63b3-1jammy [1,526 kB] 2024-04-25T07:01:00.342 INFO:teuthology.orchestra.run.smithi031.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3227-gc37f63b3-1jammy [6,257 kB] 2024-04-25T07:01:00.586 INFO:teuthology.orchestra.run.smithi031.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3227-gc37f63b3-1jammy [22.8 MB] 2024-04-25T07:01:01.415 INFO:teuthology.orchestra.run.smithi031.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3227-gc37f63b3-1jammy [14.1 kB] 2024-04-25T07:01:01.416 INFO:teuthology.orchestra.run.smithi031.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [870 kB] 2024-04-25T07:01:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3227-gc37f63b3-1jammy [2,437 kB] 2024-04-25T07:01:01.479 INFO:teuthology.orchestra.run.smithi031.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3227-gc37f63b3-1jammy [791 kB] 2024-04-25T07:01:01.489 INFO:teuthology.orchestra.run.smithi031.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3227-gc37f63b3-1jammy [148 kB] 2024-04-25T07:01:01.491 INFO:teuthology.orchestra.run.smithi031.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3227-gc37f63b3-1jammy [2,297 kB] 2024-04-25T07:01:01.532 INFO:teuthology.orchestra.run.smithi031.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3227-gc37f63b3-1jammy [8,625 kB] 2024-04-25T07:01:01.809 INFO:teuthology.orchestra.run.smithi031.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3227-gc37f63b3-1jammy [14.2 kB] 2024-04-25T07:01:01.810 INFO:teuthology.orchestra.run.smithi031.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3227-gc37f63b3-1jammy [51.6 MB] 2024-04-25T07:01:03.726 INFO:teuthology.orchestra.run.smithi031.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T07:01:03.729 INFO:teuthology.orchestra.run.smithi031.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3227-gc37f63b3-1jammy [39.0 kB] 2024-04-25T07:01:03.730 INFO:teuthology.orchestra.run.smithi031.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3227-gc37f63b3-1jammy [13.6 MB] 2024-04-25T07:01:04.110 INFO:teuthology.orchestra.run.smithi031.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [91.4 kB] 2024-04-25T07:01:04.694 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 181 MB in 6s (31.3 MB/s) 2024-04-25T07:01:04.859 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T07:01:04.961 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T07:01:04.967 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T07:01:05.048 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T07:01:05.385 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T07:01:05.412 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T07:01:05.460 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T07:01:05.738 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T07:01:05.765 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T07:01:05.804 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T07:01:06.241 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T07:01:06.268 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:01:06.316 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:01:06.927 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T07:01:06.954 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:01:06.994 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:01:07.280 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T07:01:07.308 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:01:07.347 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:01:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T07:01:07.735 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T07:01:07.774 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T07:01:08.085 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librados2. 2024-04-25T07:01:08.113 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../007-librados2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:08.152 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:08.596 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librbd1. 2024-04-25T07:01:08.623 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../008-librbd1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:08.662 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:09.065 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T07:01:09.081 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:09.115 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:09.368 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rados. 2024-04-25T07:01:09.395 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:09.434 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:09.754 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T07:01:09.781 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:09.820 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:10.064 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T07:01:10.080 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:10.122 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:10.400 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T07:01:10.427 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:10.466 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:10.811 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T07:01:10.838 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T07:01:10.877 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T07:01:11.397 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T07:01:11.424 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T07:01:11.464 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T07:01:11.724 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T07:01:11.751 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:11.791 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:12.085 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T07:01:12.112 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T07:01:12.143 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T07:01:12.396 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T07:01:12.423 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T07:01:12.462 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T07:01:12.765 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T07:01:12.781 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T07:01:12.823 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T07:01:13.143 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua5.1. 2024-04-25T07:01:13.170 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T07:01:13.210 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T07:01:13.479 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua-any. 2024-04-25T07:01:13.495 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-25T07:01:13.529 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T07:01:13.765 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package zip. 2024-04-25T07:01:13.781 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-25T07:01:13.823 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T07:01:14.135 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package unzip. 2024-04-25T07:01:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T07:01:14.238 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T07:01:14.596 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package luarocks. 2024-04-25T07:01:14.623 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T07:01:14.663 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T07:01:15.045 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librgw2. 2024-04-25T07:01:15.072 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../025-librgw2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:15.111 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:15.673 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T07:01:15.699 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:15.739 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:16.025 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T07:01:16.052 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T07:01:16.083 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T07:01:16.395 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T07:01:16.421 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:16.461 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:16.906 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-common. 2024-04-25T07:01:16.933 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:16.972 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:18.327 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-base. 2024-04-25T07:01:18.354 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:18.412 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:18.914 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T07:01:18.930 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T07:01:18.964 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T07:01:19.259 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T07:01:19.286 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T07:01:19.325 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T07:01:19.620 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T07:01:19.635 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T07:01:19.669 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T07:01:19.939 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T07:01:19.966 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T07:01:20.005 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T07:01:20.308 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T07:01:20.335 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T07:01:20.375 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T07:01:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T07:01:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T07:01:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T07:01:20.971 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-portend. 2024-04-25T07:01:20.998 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-25T07:01:21.038 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T07:01:21.324 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T07:01:21.351 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T07:01:21.394 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T07:01:21.818 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T07:01:21.846 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T07:01:21.885 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T07:01:22.222 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T07:01:22.249 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T07:01:22.280 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T07:01:22.558 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T07:01:22.581 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T07:01:22.624 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T07:01:22.927 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T07:01:22.954 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T07:01:22.993 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T07:01:23.296 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-mako. 2024-04-25T07:01:23.323 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T07:01:23.362 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T07:01:23.699 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T07:01:23.726 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T07:01:23.765 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T07:01:24.010 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T07:01:24.037 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T07:01:24.068 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T07:01:24.363 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-webob. 2024-04-25T07:01:24.389 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T07:01:24.429 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T07:01:24.774 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T07:01:24.800 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T07:01:24.843 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T07:01:25.118 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T07:01:25.145 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T07:01:25.176 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T07:01:25.395 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-paste. 2024-04-25T07:01:25.411 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T07:01:25.445 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T07:01:25.765 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T07:01:25.781 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T07:01:25.823 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T07:01:26.134 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T07:01:26.150 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T07:01:26.184 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T07:01:26.428 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T07:01:26.455 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T07:01:26.495 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T07:01:26.823 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T07:01:26.850 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T07:01:26.889 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T07:01:27.252 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T07:01:27.279 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T07:01:27.318 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T07:01:27.638 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-idna. 2024-04-25T07:01:27.665 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-25T07:01:27.704 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T07:01:27.991 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T07:01:28.018 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T07:01:28.057 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T07:01:28.335 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-requests. 2024-04-25T07:01:28.359 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T07:01:28.393 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T07:01:28.663 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T07:01:28.690 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T07:01:28.721 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T07:01:28.999 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T07:01:29.015 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:29.049 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:29.412 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T07:01:29.439 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:29.478 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:29.772 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T07:01:29.799 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T07:01:29.838 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T07:01:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T07:01:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T07:01:30.174 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T07:01:30.511 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T07:01:30.538 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:30.577 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:30.972 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T07:01:30.999 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:31.038 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T07:01:31.619 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:31.850 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:33.312 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph. 2024-04-25T07:01:33.328 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../066-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:33.370 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:33.631 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T07:01:33.657 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:33.710 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:34.067 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T07:01:34.094 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:34.133 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:34.528 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package cephadm. 2024-04-25T07:01:34.555 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../069-cephadm_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:34.594 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:34.905 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T07:01:34.932 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T07:01:34.971 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T07:01:35.258 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T07:01:35.274 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T07:01:35.316 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T07:01:36.312 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-babel. 2024-04-25T07:01:36.340 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T07:01:36.378 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T07:01:36.683 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T07:01:36.711 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T07:01:36.750 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T07:01:37.253 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T07:01:37.280 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:37.698 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T07:01:37.725 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T07:01:37.764 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T07:01:38.067 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-routes. 2024-04-25T07:01:38.094 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T07:01:38.134 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T07:01:38.470 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T07:01:38.497 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:38.536 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:39.684 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T07:01:39.711 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T07:01:39.750 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T07:01:40.245 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T07:01:40.273 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T07:01:40.312 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T07:01:40.665 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T07:01:40.692 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T07:01:40.731 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T07:01:41.035 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T07:01:41.062 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T07:01:41.101 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T07:01:41.851 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T07:01:41.880 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:42.003 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:42.973 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T07:01:43.001 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T07:01:43.040 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T07:01:43.360 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T07:01:43.387 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T07:01:43.426 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T07:01:43.755 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T07:01:43.782 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T07:01:43.821 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T07:01:44.132 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T07:01:44.160 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-25T07:01:44.199 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T07:01:44.519 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T07:01:44.547 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T07:01:44.586 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T07:01:44.889 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T07:01:45.013 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T07:01:45.013 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T07:01:45.276 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T07:01:45.303 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T07:01:45.342 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T07:01:45.662 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T07:01:45.690 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T07:01:45.728 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T07:01:46.367 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T07:01:46.393 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:46.434 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:46.778 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T07:01:46.806 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T07:01:46.845 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T07:01:47.181 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T07:01:47.198 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T07:01:47.239 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T07:01:47.676 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package jq. 2024-04-25T07:01:47.704 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T07:01:47.742 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T07:01:48.003 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package socat. 2024-04-25T07:01:48.031 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T07:01:48.070 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T07:01:48.373 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T07:01:48.389 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T07:01:48.431 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T07:01:48.786 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-test. 2024-04-25T07:01:48.814 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:48.852 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:51.090 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T07:01:51.118 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:01:51.156 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:51.468 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T07:01:51.497 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:51.535 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:51.846 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T07:01:51.874 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T07:01:51.912 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T07:01:52.342 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T07:01:52.371 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T07:01:52.400 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T07:01:52.780 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T07:01:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T07:01:52.846 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T07:01:53.285 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package pkg-config. 2024-04-25T07:01:53.313 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T07:01:53.351 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T07:01:53.604 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T07:01:53.621 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T07:01:53.662 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T07:01:54.065 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T07:01:54.093 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T07:01:54.131 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T07:01:54.401 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T07:01:54.430 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T07:01:54.468 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T07:01:54.788 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T07:01:54.816 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T07:01:54.854 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T07:01:55.165 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T07:01:55.194 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T07:01:55.232 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T07:01:55.585 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-py. 2024-04-25T07:01:55.614 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-25T07:01:55.652 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T07:01:55.980 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T07:01:56.009 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T07:01:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T07:01:56.534 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T07:01:56.563 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T07:01:56.600 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T07:01:56.903 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-toml. 2024-04-25T07:01:56.933 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-25T07:01:56.970 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T07:01:57.239 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T07:01:57.268 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T07:01:57.306 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T07:01:57.643 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T07:01:57.672 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T07:01:57.709 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T07:01:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T07:01:57.995 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T07:01:58.037 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T07:01:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package radosgw. 2024-04-25T07:01:58.711 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../116-radosgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:58.740 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:59.552 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T07:01:59.581 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:01:59.619 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:01:59.915 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T07:02:00.174 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T07:02:00.300 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T07:02:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T07:02:00.921 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T07:02:01.367 INFO:teuthology.orchestra.run.smithi031.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T07:02:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:01.918 INFO:teuthology.orchestra.run.smithi031.stdout:Adding system user cephadm....done 2024-04-25T07:02:02.066 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T07:02:02.368 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T07:02:02.619 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T07:02:02.728 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T07:02:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T07:02:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T07:02:03.365 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T07:02:03.683 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T07:02:04.052 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T07:02:04.303 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T07:02:04.571 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T07:02:04.805 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T07:02:05.056 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T07:02:05.324 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T07:02:05.584 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:05.843 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T07:02:05.969 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T07:02:06.087 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T07:02:06.196 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T07:02:06.439 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T07:02:06.833 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T07:02:06.862 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T07:02:06.926 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T07:02:07.027 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T07:02:07.378 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T07:02:07.487 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T07:02:07.747 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T07:02:08.069 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T07:02:08.690 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T07:02:08.816 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T07:02:09.519 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T07:02:09.786 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T07:02:09.912 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T07:02:10.038 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T07:02:10.348 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T07:02:10.749 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T07:02:11.008 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T07:02:11.418 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T07:02:11.711 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T07:02:12.054 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T07:02:12.355 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T07:02:12.481 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:12.799 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T07:02:13.126 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T07:02:14.339 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T07:02:14.583 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:02:14.726 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T07:02:14.993 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T07:02:15.111 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T07:02:15.237 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T07:02:15.555 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T07:02:15.814 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T07:02:16.066 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:02:16.192 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T07:02:16.510 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T07:02:16.761 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T07:02:17.020 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T07:02:17.263 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T07:02:17.531 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T07:02:17.807 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T07:02:17.932 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T07:02:18.209 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T07:02:18.335 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T07:02:18.628 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T07:02:18.929 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T07:02:19.172 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T07:02:19.297 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T07:02:19.423 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T07:02:19.512 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T07:02:19.801 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T07:02:19.926 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T07:02:20.320 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T07:02:20.571 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T07:02:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T07:02:20.948 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:02:21.074 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T07:02:21.400 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T07:02:21.701 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T07:02:21.961 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T07:02:22.087 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T07:02:22.354 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T07:02:22.639 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T07:02:23.001 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T07:02:23.377 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T07:02:23.679 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T07:02:24.031 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T07:02:24.157 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:24.286 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T07:02:24.714 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:24.840 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:24.966 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T07:02:25.243 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:25.369 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:25.495 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:25.721 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T07:02:25.721 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T07:02:26.204 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:26.330 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:26.456 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:26.582 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T07:02:27.902 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:28.028 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:28.154 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:28.280 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:28.406 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:28.915 INFO:teuthology.orchestra.run.smithi031.stdout:Adding group ceph....done 2024-04-25T07:02:29.284 INFO:teuthology.orchestra.run.smithi031.stdout:Adding system user ceph....done 2024-04-25T07:02:29.453 INFO:teuthology.orchestra.run.smithi031.stdout:Setting system user ceph properties....done 2024-04-25T07:02:29.468 INFO:teuthology.orchestra.run.smithi031.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T07:02:29.610 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T07:02:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T07:02:30.415 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:30.550 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:31.466 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T07:02:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T07:02:31.944 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:32.301 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T07:02:32.804 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:33.036 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T07:02:33.036 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T07:02:33.572 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:33.793 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T07:02:33.793 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T07:02:34.342 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:34.635 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T07:02:34.635 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T07:02:35.177 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:35.278 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:35.413 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:35.631 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T07:02:35.631 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T07:02:36.122 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:36.415 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:36.533 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:36.667 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:02:37.011 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T07:02:37.120 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T07:02:37.273 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T07:02:38.398 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T07:02:39.057 INFO:teuthology.orchestra.run.smithi031.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T07:02:39.062 DEBUG:teuthology.parallel:result is None 2024-04-25T07:02:39.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T07:02:39.354 DEBUG:teuthology.orchestra.run.smithi031:> dpkg-query -W -f '${Version}' ceph 2024-04-25T07:02:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:02:39.375 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:02:39.375 INFO:teuthology.task.install:The correct ceph version 19.0.0-3227-gc37f63b3-1jammy is installed. 2024-04-25T07:02:39.377 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T07:02:39.377 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T07:02:39.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T07:02:39.430 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T07:02:39.431 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T07:02:39.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T07:02:39.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T07:02:39.541 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T07:02:39.541 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T07:02:39.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T07:02:39.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T07:02:39.645 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T07:02:39.646 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T07:02:39.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T07:02:39.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T07:02:39.761 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T07:02:39.776 INFO:tasks.workunit:Pulling workunits from ref a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T07:02:39.776 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T07:02:39.776 INFO:tasks.workunit:timeout=3h 2024-04-25T07:02:39.776 INFO:tasks.workunit:cleanup=True 2024-04-25T07:02:39.776 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T07:02:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T07:02:39.814 INFO:teuthology.orchestra.run.smithi031.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-25T07:02:39.815 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T07:02:39.862 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-25T07:02:39.862 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-25T07:02:39.910 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/NitzanMordhai/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T07:02:39.960 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to 'a08304379f921e5e229e92c75ede8800059313ff'. 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.352 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:03:25.353 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at a08304379f9 Merge pull request #56420 from rzarzynski/wip-tools-rados-get-with-offset 2024-04-25T07:03:25.354 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T07:03:25.403 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T07:03:25.403 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T07:03:25.450 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-04-25T07:03:25.450 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-04-25T07:03:25.450 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-04-25T07:03:25.514 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:03:25.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/bad-inc-map 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:55873 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:55873 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-04-25T07:03:25.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-04-25T07:03:25.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none ' 2024-04-25T07:03:25.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:55873 ' 2024-04-25T07:03:25.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-04-25T07:03:25.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-04-25T07:03:25.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:03:25.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:03:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:03:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:03:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:03:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:03:25.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:03:25.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:03:25.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:03:25.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:03:25.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:03:25.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:03:25.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:03:25.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:03:25.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:03:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:03:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:03:25.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:03:25.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20195 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:03:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-04-25T07:03:25.562 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:03:25.562 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:25.562 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:25.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20195 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:03:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:03:25.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-04-25T07:03:25.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-04-25T07:03:26.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:03:26.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:03:26.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:03:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:03:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:26.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:03:26.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:03:26.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:03:26.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:03:26.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:03:26.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:03:26.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:03:26.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20195/ceph-mon.a.asok 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:03:26.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20195/ceph-mon.a.asok config get fsid 2024-04-25T07:03:26.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:03:26.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:03:26.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:03:26.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:03:26.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20195/ceph-mon.a.asok 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:03:26.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20195/ceph-mon.a.asok config get mon_host 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-04-25T07:03:26.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:03:27.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:03:27.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:03:27.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:03:27.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:03:27.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:27.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:27.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:03:27.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:03:27.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 ' 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:03:27.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:27.636 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:03:27.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-04-25T07:03:27.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:03:27.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=569a414a-c6b0-4c47-8777-6163e02a5b18 2024-04-25T07:03:27.638 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 569a414a-c6b0-4c47-8777-6163e02a5b18 2024-04-25T07:03:27.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 569a414a-c6b0-4c47-8777-6163e02a5b18' 2024-04-25T07:03:27.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:03:27.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/ACpmsVkpJxAA4GDsInp8O0MqrLpI3Ddu1w== 2024-04-25T07:03:27.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/ACpmsVkpJxAA4GDsInp8O0MqrLpI3Ddu1w=="}' 2024-04-25T07:03:27.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 569a414a-c6b0-4c47-8777-6163e02a5b18 -i td/bad-inc-map/0/new.json 2024-04-25T07:03:27.945 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:03:27.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-04-25T07:03:27.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/ACpmsVkpJxAA4GDsInp8O0MqrLpI3Ddu1w== --osd-uuid 569a414a-c6b0-4c47-8777-6163e02a5b18 2024-04-25T07:03:28.053 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.051+0000 7fef6bb3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:28.055 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.055+0000 7fef6bb3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:28.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.055+0000 7fef6bb3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:28.057 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.055+0000 7fef6bb3e7c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:03:28.057 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.055+0000 7fef6bb3e7c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:03:28.057 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.055+0000 7fef6bb3e7c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:03:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:28.175+0000 7fef6bb3e7c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-04-25T07:03:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-04-25T07:03:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:03:33.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:03:33.968 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:03:33.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:03:34.223 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:03:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:03:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:03:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:03:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:03:34.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:03:34.255 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:34.255+0000 7f441dfa07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:34.258 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:34.259+0000 7f441dfa07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:34.259+0000 7f441dfa07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:03:34.451 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:03:34.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:34.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:34.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:34.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:34.823+0000 7f441dfa07c0 -1 Falling back to public interface 2024-04-25T07:03:35.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:35.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:35.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:03:35.675 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:03:35.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:35.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:35.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:36.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:36.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:36.973 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:03:36.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:03:36.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:36.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:37.265 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:03:37.263+0000 7f441dfa07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:03:37.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:38.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:38.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:38.437 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:03:38.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:03:38.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:38.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:39.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:40.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:40.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:40.112 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:03:40.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:03:40.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:40.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:40.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:41.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:41.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:41.576 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:03:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:03:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:42.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:43.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:43.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:43.138 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:03:43.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:03:43.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:43.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:43.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:44.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:44.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:44.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:03:44.565 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:03:44.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:44.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:45.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:46.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:46.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:46.104 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:03:46.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:03:46.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:46.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:46.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:47.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:47.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:47.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:03:47.537 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:03:47.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:47.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:48.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:03:49.008 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:03:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:49.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:50.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:50.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:50.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:03:50.837 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:03:50.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:50.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:51.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:52.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:52.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:52.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:03:52.834 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:03:52.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:52.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:53.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:54.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:54.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:54.847 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:03:54.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:03:54.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:54.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:55.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:56.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:56.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:56.456 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:03:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:03:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:57.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:58.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:03:58.285 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:03:58.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:58.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:59.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:00.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:00.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:00.369 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:04:00.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:04:00.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:00.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:04:01.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:02.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:02.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:02.220 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:04:02.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:04:02.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:02.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:04:02.667 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/280470145,v1:127.0.0.1:6801/280470145] [v2:127.0.0.1:6802/280470145,v1:127.0.0.1:6803/280470145] exists,up 569a414a-c6b0-4c47-8777-6163e02a5b18 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 ' 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:04:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:04:02.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:04:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-04-25T07:04:02.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:04:02.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43b1cb3b-f1a9-42ec-8af4-419947d60ce6 2024-04-25T07:04:02.674 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 43b1cb3b-f1a9-42ec-8af4-419947d60ce6 2024-04-25T07:04:02.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 43b1cb3b-f1a9-42ec-8af4-419947d60ce6' 2024-04-25T07:04:02.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:04:02.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiACpmlK95KRAAKtRn6JHBqji6tP9TvP5oJQ== 2024-04-25T07:04:02.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiACpmlK95KRAAKtRn6JHBqji6tP9TvP5oJQ=="}' 2024-04-25T07:04:02.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43b1cb3b-f1a9-42ec-8af4-419947d60ce6 -i td/bad-inc-map/1/new.json 2024-04-25T07:04:03.185 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:04:03.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-04-25T07:04:03.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiACpmlK95KRAAKtRn6JHBqji6tP9TvP5oJQ== --osd-uuid 43b1cb3b-f1a9-42ec-8af4-419947d60ce6 2024-04-25T07:04:03.231 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.231+0000 7f9d0be837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:03.234 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.235+0000 7f9d0be837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:03.236 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.235+0000 7f9d0be837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:03.236 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.235+0000 7f9d0be837c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:04:03.236 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.235+0000 7f9d0be837c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:04:03.237 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.235+0000 7f9d0be837c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:04:03.265 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:03.267+0000 7f9d0be837c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-04-25T07:04:09.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-04-25T07:04:09.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:04:09.075 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:04:09.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:04:09.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:04:09.652 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:04:09.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:04:09.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:04:09.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:04:09.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:04:09.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:04:09.682 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:09.683+0000 7fb1a1ac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:09.686 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:09.687+0000 7fb1a1ac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:09.687 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:09.687+0000 7fb1a1ac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:10.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:04:10.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:10.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:10.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:10.254 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:10.255+0000 7fb1a1ac97c0 -1 Falling back to public interface 2024-04-25T07:04:10.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:11.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:11.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:11.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:04:11.563 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:04:11.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:11.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:11.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:12.851 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:12.851+0000 7fb1a1ac97c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:04:12.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:12.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:12.995 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:04:12.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:04:12.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:12.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:13.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:14.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:14.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:14.570 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:04:14.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:04:14.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:14.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:15.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:16.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:16.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:16.136 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:04:16.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:04:16.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:16.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:16.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:17.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:17.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:17.676 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:04:17.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:04:17.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:17.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:18.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:19.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:19.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:19.142 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:04:19.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:04:19.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:19.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:19.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:20.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:20.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:20.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:04:20.683 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:04:20.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:20.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:21.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:22.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:22.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:22.141 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:04:22.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:04:22.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:22.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:22.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:23.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:23.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:23.687 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:04:23.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:04:23.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:23.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:24.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:25.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:25.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:25.130 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:04:25.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:04:25.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:25.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:25.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:26.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:26.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:26.881 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:04:26.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:04:26.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:26.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:27.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:28.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:28.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:28.624 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:04:28.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:04:28.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:28.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:29.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:30.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:30.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:30.312 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:04:30.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:04:30.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:30.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:31.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:32.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:32.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:32.484 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:04:32.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:04:32.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:32.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:33.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:34.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:34.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:34.389 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:04:34.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:04:34.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:34.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:34.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:35.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:35.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:35.965 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:04:35.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:04:35.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:35.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:36.433 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4273762680,v1:127.0.0.1:6811/4273762680] [v2:127.0.0.1:6812/4273762680,v1:127.0.0.1:6813/4273762680] exists,up 43b1cb3b-f1a9-42ec-8af4-419947d60ce6 2024-04-25T07:04:36.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 ' 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:04:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:36.438 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:04:36.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:04:36.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:04:36.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-04-25T07:04:36.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:04:36.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca515ecd-dd73-4a2f-923f-aa859f9e3dc6 2024-04-25T07:04:36.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ca515ecd-dd73-4a2f-923f-aa859f9e3dc6' 2024-04-25T07:04:36.446 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 ca515ecd-dd73-4a2f-923f-aa859f9e3dc6 2024-04-25T07:04:36.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:04:36.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEACpmj6IBHBAAZp/BMFOMOHoi7WzMXJfrWg== 2024-04-25T07:04:36.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEACpmj6IBHBAAZp/BMFOMOHoi7WzMXJfrWg=="}' 2024-04-25T07:04:36.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca515ecd-dd73-4a2f-923f-aa859f9e3dc6 -i td/bad-inc-map/2/new.json 2024-04-25T07:04:37.020 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:04:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-04-25T07:04:37.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEACpmj6IBHBAAZp/BMFOMOHoi7WzMXJfrWg== --osd-uuid ca515ecd-dd73-4a2f-923f-aa859f9e3dc6 2024-04-25T07:04:37.073 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.070+0000 7f39f246b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:37.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.098+0000 7f39f246b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:37.101 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.102+0000 7f39f246b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:37.102 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.102+0000 7f39f246b7c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:04:37.102 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.102+0000 7f39f246b7c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:04:37.102 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.102+0000 7f39f246b7c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:04:37.202 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:37.202+0000 7f39f246b7c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-04-25T07:04:42.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-04-25T07:04:42.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:04:42.807 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:04:42.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:04:42.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:04:43.373 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:04:43.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:04:43.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=baa4b6cb-1ef7-4a74-b907-46bbaf3ba6c1 --auth-supported=none --mon-host=127.0.0.1:55873 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:04:43.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:04:43.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:04:43.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:04:43.397 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:43.393+0000 7f099a68a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:43.397+0000 7f099a68a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:43.400 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:43.397+0000 7f099a68a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:43.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:04:43.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:43.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:04:43.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:43.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:43.800 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:04:43.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:43.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:43.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:43.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:43.963 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:43.961+0000 7f099a68a7c0 -1 Falling back to public interface 2024-04-25T07:04:44.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:45.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:45.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:45.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:04:45.234 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:04:45.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:45.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:45.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:46.540 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:04:46.537+0000 7f099a68a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:04:46.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:46.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:46.651 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:04:46.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:04:46.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:46.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:47.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:48.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:48.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:48.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:04:48.200 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:04:48.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:48.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:48.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:49.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:49.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:49.644 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:04:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:04:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:50.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:51.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:51.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:51.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:04:51.162 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:04:51.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:51.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:51.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:52.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:52.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:52.706 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:04:52.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:04:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:54.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:54.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:54.159 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:04:54.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:04:54.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:54.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:54.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:04:55.685 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:04:55.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:55.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:56.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:04:57.124 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:04:57.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:57.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:57.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:04:58.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:04:58.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:58.670 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:04:58.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:04:58.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:58.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:59.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:00.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:00.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:00.709 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:05:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:05:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:01.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:02.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:02.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:02.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:05:02.658 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:05:02.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:02.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:03.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:04.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:04.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:04.588 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:05:04.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:05:04.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:04.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:05.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:06.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:06.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:06.597 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:05:06.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:05:06.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:06.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:07.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:08.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:08.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:08.593 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:05:08.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:05:08.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:08.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:09.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:10.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:10.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:10.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:05:10.107 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:05:10.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:10.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:10.573 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2776701943,v1:127.0.0.1:6819/2776701943] [v2:127.0.0.1:6820/2776701943,v1:127.0.0.1:6821/2776701943] exists,up ca515ecd-dd73-4a2f-923f-aa859f9e3dc6 2024-04-25T07:05:10.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:05:10.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:05:10.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:05:10.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-04-25T07:05:11.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-04-25T07:05:11.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-04-25T07:05:11.780 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T07:05:11.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:05:12.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-04-25T07:05:14.110 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 1 2024-04-25T07:05:14.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-04-25T07:05:15.153 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 2 2024-04-25T07:05:15.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:20.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:05:20.184 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:05:20.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:20.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:20.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:20.625 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/280470145,v1:127.0.0.1:6801/280470145] [v2:127.0.0.1:6802/280470145,v1:127.0.0.1:6803/280470145] exists,up 569a414a-c6b0-4c47-8777-6163e02a5b18 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:05:20.627 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:05:20.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:20.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:20.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:05:21.020 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4273762680,v1:127.0.0.1:6811/4273762680] [v2:127.0.0.1:6812/4273762680,v1:127.0.0.1:6813/4273762680] exists,up 43b1cb3b-f1a9-42ec-8af4-419947d60ce6 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:05:21.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:21.023 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:05:21.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:21.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:05:21.480 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2776701943,v1:127.0.0.1:6819/2776701943] [v2:127.0.0.1:6820/2776701943,v1:127.0.0.1:6821/2776701943] exists,up ca515ecd-dd73-4a2f-923f-aa859f9e3dc6 2024-04-25T07:05:21.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:05:21.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:05:21.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:05:21.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:11.529+0000 7f098b62e640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:12.253+0000 7f098b62e640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:12.593+0000 7f098b62e640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:13.553+0000 7f098962a640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:13.953+0000 7f098962a640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:14.057+0000 7f098962a640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:14.157+0000 7f098b62e640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:14.885+0000 7f098962a640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:15.205+0000 7f098b62e640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:15.209+0000 7f098b62e640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:15.573+0000 7f098b62e640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-04-25T07:05:21.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:11.529+0000 7f098b62e640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:12.253+0000 7f098b62e640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:12.593+0000 7f098b62e640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:13.553+0000 7f098962a640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:13.957+0000 7f098962a640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:14.057+0000 7f098962a640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-04-25T07:05:21.486 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:14.157+0000 7f098b62e640 10 osd.2 21 handle_osd_map bailing because last < start (22<23) 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:14.889+0000 7f098962a640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:15.205+0000 7f098b62e640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:15.209+0000 7f098b62e640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:05:15.573+0000 7f098b62e640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stdout:success 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-04-25T07:05:21.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-04-25T07:05:21.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-04-25T07:05:21.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-04-25T07:05:22.048 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' does not exist 2024-04-25T07:05:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-04-25T07:05:22.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:22.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:22.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:22.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:22.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:27.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:27.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-04-25T07:05:27.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:05:27.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:05:27.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:05:27.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:27.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:27.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:27.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:27.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:27.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:27.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:27.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:27.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:27.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:27.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:27.394 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:27.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:27.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:27.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:27.395 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:27.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:27.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:05:27.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:05:27.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:27.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:27.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:05:27.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20195 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/bad-inc-map 0 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:27.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:27.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:27.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:27.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:27.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:27.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:27.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:27.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:27.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:27.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:27.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:27.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:27.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:27.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:05:27.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:05:27.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:27.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:27.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20195 2024-04-25T07:05:27.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20195 2024-04-25T07:05:27.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:27.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:27.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:05:27.427 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:05:27.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:05:27.482 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-04-25T07:05:27.482 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-04-25T07:05:27.537 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:05:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/divergent-priors 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-04-25T07:05:27.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-04-25T07:05:27.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-04-25T07:05:27.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-04-25T07:05:27.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-04-25T07:05:27.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-04-25T07:05:27.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-04-25T07:05:27.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-04-25T07:05:27.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-04-25T07:05:27.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-04-25T07:05:27.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-04-25T07:05:27.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-04-25T07:05:27.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none ' 2024-04-25T07:05:27.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T07:05:27.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T07:05:27.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-04-25T07:05:27.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:05:27.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-04-25T07:05:27.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_divergent_2 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_divergent_3 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_divergent_ec' 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:05:27.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:05:27.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:05:27.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:27.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:27.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:27.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:27.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:27.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:27.558 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:27.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:27.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:27.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:27.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:27.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:27.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:27.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:27.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:27.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:27.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:27.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:05:27.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:05:27.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:27.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:27.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:05:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:05:27.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:05:27.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:27.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:27.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.25959 2024-04-25T07:05:27.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-04-25T07:05:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-04-25T07:05:27.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:05:27.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:05:27.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:05:28.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:05:28.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:28.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:28.096 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:28.096 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:28.096 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:28.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:05:28.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:05:28.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:05:28.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:05:28.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:28.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:28.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:05:28.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:05:28.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:28.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:28.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:28.373 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:28.373 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:28.373 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:28.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:05:28.374 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:28.374 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get fsid 2024-04-25T07:05:28.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:05:28.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:28.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:28.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:05:28.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:05:28.493 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:28.493 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:28.493 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:28.493 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:28.493 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:05:28.494 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:28.494 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get mon_host 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:05:28.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:05:29.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:05:29.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:29.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:29.520 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:29.520 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:29.520 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:29.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:05:29.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:29.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:05:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-04-25T07:05:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:05:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:05:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:29.588 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:05:29.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:05:29.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:05:29.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:05:29.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:05:29.590 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:05:29.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0a9b5c4-c2ec-465b-90bd-285d2f6545eb 2024-04-25T07:05:29.590 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 d0a9b5c4-c2ec-465b-90bd-285d2f6545eb 2024-04-25T07:05:29.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d0a9b5c4-c2ec-465b-90bd-285d2f6545eb' 2024-04-25T07:05:29.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:05:29.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5ACpm+PGnJBAA6ngEMwNOeDrdBJt7G6n74w== 2024-04-25T07:05:29.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5ACpm+PGnJBAA6ngEMwNOeDrdBJt7G6n74w=="}' 2024-04-25T07:05:29.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0a9b5c4-c2ec-465b-90bd-285d2f6545eb -i td/divergent-priors/0/new.json 2024-04-25T07:05:29.907 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:05:29.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:05:29.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5ACpm+PGnJBAA6ngEMwNOeDrdBJt7G6n74w== --osd-uuid d0a9b5c4-c2ec-465b-90bd-285d2f6545eb 2024-04-25T07:05:29.976 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:29.972+0000 7fa72936d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:29.979 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:29.976+0000 7fa72936d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:29.980 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:29.976+0000 7fa72936d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:29.981 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:29.976+0000 7fa72936d7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:05:29.981 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:29.976+0000 7fa72936d7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:05:29.981 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:29.976+0000 7fa72936d7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:05:30.024 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:30.020+0000 7fa72936d7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:05:36.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:05:36.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:05:36.083 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:05:36.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:05:36.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:05:36.712 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:05:36.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:05:36.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:05:36.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:05:36.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:05:36.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:05:36.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:36.752+0000 7f98007be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:36.757 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:36.752+0000 7f98007be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:36.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:36.756+0000 7f98007be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:37.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:05:37.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:37.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:05:37.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:37.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:37.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:37.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:37.145 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:05:37.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:37.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:37.340 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:37.336+0000 7f98007be7c0 -1 Falling back to public interface 2024-04-25T07:05:37.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:38.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:38.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:38.609 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:05:38.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:05:38.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:38.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:39.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:39.891 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:05:39.888+0000 7f98007be7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:05:40.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:40.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:40.083 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:05:40.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:05:40.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:40.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:41.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:41.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:41.662 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:05:41.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:05:41.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:41.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:42.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:43.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:43.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:43.104 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:05:43.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:05:43.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:43.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:43.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:44.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:44.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:44.677 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:05:44.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:05:44.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:44.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:45.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:46.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:46.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:46.132 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:05:46.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:05:46.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:46.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:46.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:47.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:47.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:47.626 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:05:47.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:05:47.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:47.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:48.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:49.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:49.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:49.085 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:05:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:05:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:49.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:50.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:50.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:50.652 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:05:50.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:05:50.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:50.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:51.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:52.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:52.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:52.193 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:05:52.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:05:52.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:52.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:52.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:53.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:53.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:53.742 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:05:53.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:05:53.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:53.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:54.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:55.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:55.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:55.587 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:05:55.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:05:55.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:55.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:56.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:57.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:57.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:57.393 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:05:57.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:05:57.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:57.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:59.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:59.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:59.103 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:05:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:05:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:00.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:01.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:01.171 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:06:01.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:01.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:06:01.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:01.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:01.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:02.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:02.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:02.739 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:06:02.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:06:02.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:02.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:03.216 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3276745302,v1:127.0.0.1:6803/3276745302] [v2:127.0.0.1:6804/3276745302,v1:127.0.0.1:6805/3276745302] exists,up d0a9b5c4-c2ec-465b-90bd-285d2f6545eb 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:06:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:06:03.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:06:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:06:03.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:06:03.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f0287046-382c-44ba-a3ab-51ca072bda43 2024-04-25T07:06:03.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f0287046-382c-44ba-a3ab-51ca072bda43' 2024-04-25T07:06:03.222 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 f0287046-382c-44ba-a3ab-51ca072bda43 2024-04-25T07:06:03.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:06:03.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbACpm9BfxDhAAKwQMEUZt4AkNOgt91BuuIw== 2024-04-25T07:06:03.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbACpm9BfxDhAAKwQMEUZt4AkNOgt91BuuIw=="}' 2024-04-25T07:06:03.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f0287046-382c-44ba-a3ab-51ca072bda43 -i td/divergent-priors/1/new.json 2024-04-25T07:06:03.744 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:06:03.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:06:03.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbACpm9BfxDhAAKwQMEUZt4AkNOgt91BuuIw== --osd-uuid f0287046-382c-44ba-a3ab-51ca072bda43 2024-04-25T07:06:03.806 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.803+0000 7fddeffb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:03.807 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.807+0000 7fddeffb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:03.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.807+0000 7fddeffb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:03.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.807+0000 7fddeffb27c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:06:03.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.807+0000 7fddeffb27c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:06:03.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.807+0000 7fddeffb27c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:06:03.846 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:03.847+0000 7fddeffb27c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:06:09.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:06:09.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:06:09.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:06:09.671 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:06:09.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:06:10.250 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:06:10.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:06:10.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:06:10.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:06:10.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:06:10.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:06:10.291 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:10.291+0000 7f2e445c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:10.337 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:10.335+0000 7f2e445c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:10.339 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:10.339+0000 7f2e445c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:10.759 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:06:10.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:06:10.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:10.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:10.907 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:10.907+0000 7f2e445c17c0 -1 Falling back to public interface 2024-04-25T07:06:11.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:12.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:12.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:12.253 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:06:12.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:12.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:12.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:12.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:13.386 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:13.383+0000 7f2e445c17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:06:13.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:13.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:13.726 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:06:13.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:13.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:13.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:14.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:15.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:15.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:15.177 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:06:15.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:15.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:15.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:15.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:16.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:16.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:16.651 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:06:16.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:16.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:16.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:17.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:18.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:18.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:18.386 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:06:18.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:18.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:18.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:18.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:19.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:19.928 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:06:19.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:19.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:19.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:20.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:21.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:21.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:21.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:21.515 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:06:21.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:21.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:21.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:22.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:22.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:22.964 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:06:22.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:22.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:22.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:23.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:24.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:24.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:24.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:24.523 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:06:24.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:24.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:25.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:26.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:26.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:26.416 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:06:26.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:26.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:26.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:27.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:28.085 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:06:28.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:28.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:28.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:28.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:29.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:29.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:29.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:06:29.755 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:06:29.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:29.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:30.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:31.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:31.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:31.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:06:31.966 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:06:31.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:31.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:32.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:33.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:33.668 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:06:33.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:33.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:06:33.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:33.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:34.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:35.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:35.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:35.472 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:06:35.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:06:35.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:35.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:35.953 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/111237492,v1:127.0.0.1:6811/111237492] [v2:127.0.0.1:6812/111237492,v1:127.0.0.1:6813/111237492] exists,up f0287046-382c-44ba-a3ab-51ca072bda43 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:06:35.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:06:35.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:06:35.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:06:35.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:06:35.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:06:35.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:06:35.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:06:35.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:06:35.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4772361e-836b-4c5f-b574-d03f08754357 2024-04-25T07:06:35.961 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 4772361e-836b-4c5f-b574-d03f08754357 2024-04-25T07:06:35.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4772361e-836b-4c5f-b574-d03f08754357' 2024-04-25T07:06:35.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:06:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7ACpm3UOmOhAAdRKFljg8Opqg2zutubPfZg== 2024-04-25T07:06:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7ACpm3UOmOhAAdRKFljg8Opqg2zutubPfZg=="}' 2024-04-25T07:06:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4772361e-836b-4c5f-b574-d03f08754357 -i td/divergent-priors/2/new.json 2024-04-25T07:06:36.606 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:06:36.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:06:36.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7ACpm3UOmOhAAdRKFljg8Opqg2zutubPfZg== --osd-uuid 4772361e-836b-4c5f-b574-d03f08754357 2024-04-25T07:06:36.662 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.659+0000 7fedc0d207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:36.686 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.687+0000 7fedc0d207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:36.688 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.687+0000 7fedc0d207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:36.688 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.687+0000 7fedc0d207c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:06:36.688 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.687+0000 7fedc0d207c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:06:36.688 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.687+0000 7fedc0d207c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:06:36.770 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:36.767+0000 7fedc0d207c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:06:42.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:06:42.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:06:42.318 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:06:42.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:06:42.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:06:42.931 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:06:42.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:06:42.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:06:42.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:06:42.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:06:42.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:06:42.973 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:42.971+0000 7f52a7a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:42.985 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:42.983+0000 7f52a7a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:42.986 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:42.983+0000 7f52a7a6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:43.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:06:43.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:06:43.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:06:43.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:06:43.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:06:43.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:43.365 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:06:43.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:06:43.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:43.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:43.548 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:43.547+0000 7f52a7a6a7c0 -1 Falling back to public interface 2024-04-25T07:06:43.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:44.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:44.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:44.845 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:06:44.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:44.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:44.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:45.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:46.043 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:06:46.042+0000 7f52a7a6a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:06:46.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:46.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:46.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:46.329 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:06:46.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:46.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:46.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:47.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:47.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:47.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:47.905 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:06:47.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:47.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:48.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:49.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:49.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:49.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:49.361 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:06:49.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:49.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:49.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:50.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:50.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:50.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:50.956 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:06:50.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:50.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:51.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:52.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:52.424 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:06:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:53.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:54.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:54.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:54.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:54.008 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:06:54.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:54.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:54.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:55.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:55.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:55.542 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:06:55.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:55.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:55.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:56.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:57.091 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:06:57.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:57.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:57.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:58.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:58.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:58.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:58.720 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:06:58.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:58.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:59.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:00.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:00.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:00.568 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:07:00.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:07:00.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:00.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:01.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:02.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:02.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:02.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:07:02.356 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:07:02.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:02.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:03.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:04.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:04.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:04.150 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:07:04.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:07:04.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:04.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:04.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:05.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:05.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:05.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:07:05.837 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:07:05.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:05.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:06.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:07.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:07.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:07.540 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:07:07.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:07:07.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:07.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:08.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:09.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:09.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:09.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:07:09.077 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:07:09.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:09.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:09.574 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2952756179,v1:127.0.0.1:6819/2952756179] [v2:127.0.0.1:6820/2952756179,v1:127.0.0.1:6821/2952756179] exists,up 4772361e-836b-4c5f-b574-d03f08754357 2024-04-25T07:07:09.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:09.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:09.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:09.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-04-25T07:07:10.230 INFO:tasks.workunit.client.0.smithi031.stderr:noout is set 2024-04-25T07:07:10.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-04-25T07:07:11.084 INFO:tasks.workunit.client.0.smithi031.stderr:noin is set 2024-04-25T07:07:11.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-04-25T07:07:11.762 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T07:07:11.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-04-25T07:07:11.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:07:12.602 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T07:07:12.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:07:13.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-04-25T07:07:14.280 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 3 2024-04-25T07:07:14.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-04-25T07:07:15.034 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 2 2024-04-25T07:07:15.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-04-25T07:07:15.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:15.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:15.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:15.667 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:07:15.667 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:07:15.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:15.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:15.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:15.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836498 2024-04-25T07:07:15.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836498 2024-04-25T07:07:15.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836498' 2024-04-25T07:07:15.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:15.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:16.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672973 2024-04-25T07:07:16.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672973 2024-04-25T07:07:16.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836498 1-42949672973' 2024-04-25T07:07:16.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:16.017 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:16.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509446 2024-04-25T07:07:16.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509446 2024-04-25T07:07:16.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836498 1-42949672973 2-64424509446' 2024-04-25T07:07:16.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:16.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836498 2024-04-25T07:07:16.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:16.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:16.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836498 2024-04-25T07:07:16.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:16.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836498 2024-04-25T07:07:16.191 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836498 2024-04-25T07:07:16.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-25T07:07:16.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-25T07:07:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:16.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672973 2024-04-25T07:07:16.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:16.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-04-25T07:07:16.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672973 2024-04-25T07:07:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-04-25T07:07:16.673 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672973 2024-04-25T07:07:16.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:17.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672973 2024-04-25T07:07:17.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:17.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509446 2024-04-25T07:07:17.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:17.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:17.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509446 2024-04-25T07:07:17.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:17.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509446 2024-04-25T07:07:17.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-25T07:07:17.166 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509446 2024-04-25T07:07:17.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:17.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509446 -lt 64424509446 2024-04-25T07:07:17.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-04-25T07:07:17.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:07:17.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:07:17.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:07:17.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:07:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:07:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:07:17.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:07:17.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:07:17.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:07:17.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:07:17.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:07:17.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:07:17.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:07:17.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:17.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:18.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:18.197 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:07:18.197 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:07:18.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:18.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:18.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:18.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836501 2024-04-25T07:07:18.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836501 2024-04-25T07:07:18.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501' 2024-04-25T07:07:18.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:18.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:18.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:07:18.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:07:18.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-04-25T07:07:18.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:18.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:18.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T07:07:18.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T07:07:18.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509449' 2024-04-25T07:07:18.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:18.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836501 2024-04-25T07:07:18.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:18.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:18.755 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836501 2024-04-25T07:07:18.755 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:18.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836501 2024-04-25T07:07:18.757 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836501 2024-04-25T07:07:18.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-04-25T07:07:18.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:19.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836501 2024-04-25T07:07:19.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:07:20.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:07:20.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:20.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836501 2024-04-25T07:07:20.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:20.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:07:20.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:20.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:20.743 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:07:20.743 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:20.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:07:20.745 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:07:20.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:07:20.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:21.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-25T07:07:21.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:21.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T07:07:21.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:21.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:21.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T07:07:21.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:21.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T07:07:21.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T07:07:21.240 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509449 2024-04-25T07:07:21.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:21.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-25T07:07:21.701 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:07:21.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:21.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:22.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:07:22.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:07:22.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:07:22.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:07:22.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:07:22.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:07:22.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:07:22.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:07:22.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:07:22.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:07:22.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:22.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:23.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:07:23.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:07:23.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:07:23.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-04-25T07:07:23.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-04-25T07:07:23.669 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:07:23.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-04-25T07:07:23.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-04-25T07:07:23.686 INFO:tasks.workunit.client.0.smithi031.stdout:primary and soon to be divergent is 1 2024-04-25T07:07:23.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-04-25T07:07:24.100 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:24.101 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:07:13.179825+0000 0'0 29:19 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:58:14.500539+0000 0 0 2024-04-25T07:07:24.101 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:07:24.101 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:24.101 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:07:24.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-04-25T07:07:24.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:07:24.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-04-25T07:07:24.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-04-25T07:07:24.119 INFO:tasks.workunit.client.0.smithi031.stdout:writing initial objects 2024-04-25T07:07:24.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-04-25T07:07:24.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-04-25T07:07:24.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-04-25T07:07:24.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-04-25T07:07:24.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-04-25T07:07:24.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-04-25T07:07:24.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-04-25T07:07:24.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-04-25T07:07:25.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-04-25T07:07:25.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-04-25T07:07:25.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-04-25T07:07:25.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-04-25T07:07:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-04-25T07:07:25.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-04-25T07:07:25.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-04-25T07:07:26.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-04-25T07:07:26.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-04-25T07:07:26.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-04-25T07:07:26.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-04-25T07:07:26.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-04-25T07:07:26.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-04-25T07:07:27.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-04-25T07:07:27.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-04-25T07:07:27.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-04-25T07:07:27.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-04-25T07:07:27.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-04-25T07:07:27.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-04-25T07:07:27.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-04-25T07:07:28.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-04-25T07:07:28.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-04-25T07:07:28.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-04-25T07:07:28.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-04-25T07:07:28.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-04-25T07:07:28.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-04-25T07:07:29.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-04-25T07:07:29.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-04-25T07:07:29.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-04-25T07:07:29.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-04-25T07:07:29.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-04-25T07:07:29.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-04-25T07:07:29.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-04-25T07:07:30.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-04-25T07:07:30.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-04-25T07:07:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-04-25T07:07:30.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-04-25T07:07:30.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-04-25T07:07:30.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-04-25T07:07:30.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-04-25T07:07:31.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-04-25T07:07:31.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-04-25T07:07:31.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-04-25T07:07:31.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-04-25T07:07:31.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-04-25T07:07:31.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-04-25T07:07:32.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-04-25T07:07:32.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-04-25T07:07:32.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-04-25T07:07:32.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-04-25T07:07:32.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-04-25T07:07:32.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-04-25T07:07:33.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-04-25T07:07:33.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-04-25T07:07:33.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-04-25T07:07:33.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-04-25T07:07:33.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-04-25T07:07:33.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-04-25T07:07:33.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-04-25T07:07:34.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-04-25T07:07:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-04-25T07:07:34.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-04-25T07:07:34.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-04-25T07:07:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-04-25T07:07:34.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-04-25T07:07:35.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-04-25T07:07:35.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-04-25T07:07:35.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-04-25T07:07:35.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-04-25T07:07:35.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-04-25T07:07:35.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-04-25T07:07:35.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-04-25T07:07:36.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-04-25T07:07:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-04-25T07:07:36.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-04-25T07:07:36.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-04-25T07:07:36.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-04-25T07:07:36.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-04-25T07:07:37.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-04-25T07:07:37.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-04-25T07:07:37.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-04-25T07:07:37.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-04-25T07:07:37.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-04-25T07:07:37.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-04-25T07:07:37.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-04-25T07:07:38.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:38.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-04-25T07:07:38.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:38.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-04-25T07:07:38.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:38.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-04-25T07:07:38.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:38.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-04-25T07:07:38.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:38.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-04-25T07:07:38.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:38.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-04-25T07:07:39.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:39.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-04-25T07:07:39.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:39.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-04-25T07:07:39.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:07:39.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T07:07:39.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:07:39.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-04-25T07:07:39.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:07:39.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:07:39.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:07:39.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:07:39.578 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:07:39.579 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:07:39.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:07:39.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:07:39.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:07:39.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:07:39.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:07:39.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:07:39.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:07:39.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:39.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:40.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:40.144 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:07:40.144 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:07:40.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:40.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:40.145 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:40.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-04-25T07:07:40.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-04-25T07:07:40.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-04-25T07:07:40.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:40.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:40.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-04-25T07:07:40.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-04-25T07:07:40.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672981' 2024-04-25T07:07:40.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:40.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T07:07:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T07:07:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672981 2-64424509455' 2024-04-25T07:07:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:40.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-04-25T07:07:40.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:40.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:40.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-04-25T07:07:40.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:40.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-04-25T07:07:40.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-04-25T07:07:40.665 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836507 2024-04-25T07:07:40.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:41.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836507 2024-04-25T07:07:41.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:07:42.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:07:42.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:42.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-04-25T07:07:42.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:42.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-04-25T07:07:42.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:42.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:42.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-04-25T07:07:42.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:42.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-04-25T07:07:42.567 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672981 2024-04-25T07:07:42.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-25T07:07:42.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:43.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672981 2024-04-25T07:07:43.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:43.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T07:07:43.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:43.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:43.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T07:07:43.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:43.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T07:07:43.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T07:07:43.113 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509455 2024-04-25T07:07:43.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:43.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-25T07:07:43.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:07:43.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:43.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:44.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:07:44.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:07:44.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:07:44.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:07:44.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:07:44.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:07:44.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:07:44.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:07:44.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:07:44.602 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:07:44.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:44.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:45.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:07:45.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:07:45.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:07:45.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-04-25T07:07:45.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:07:45.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:07:45.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:07:45.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:07:45.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-04-25T07:07:45.687 INFO:tasks.workunit.client.0.smithi031.stdout:blackholing osds 0 2 2024-04-25T07:07:45.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-04-25T07:07:45.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-04-25T07:07:46.100 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:46.100 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:07:13.179825+0000 29'100 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:58:14.500539+0000 0 0 2024-04-25T07:07:46.100 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:07:46.100 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:46.100 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:07:46.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T07:07:46.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-04-25T07:07:46.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:07:46.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:07:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:07:46.118 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:07:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T07:07:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:07:46.238 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:07:46.238 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:07:46.238 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:07:46.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T07:07:46.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-04-25T07:07:46.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:07:46.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:07:46.254 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:46.254 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:46.254 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:07:46.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:07:46.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T07:07:46.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:07:46.369 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:07:46.369 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:07:46.369 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:07:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-04-25T07:07:46.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-04-25T07:07:46.383 INFO:tasks.workunit.client.0.smithi031.stdout:writing divergent object 2024-04-25T07:07:46.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-04-25T07:07:46.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-04-25T07:07:46.384 INFO:tasks.workunit.client.0.smithi031.stdout:create missing divergent object 2024-04-25T07:07:46.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=existing_99 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/divergent-priors 2024-04-25T07:07:46.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test existing_99 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-04-25T07:07:46.387 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T07:07:46.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T07:07:46.834 INFO:tasks.workunit.client.0.smithi031.stderr:0 2024-04-25T07:07:46.834 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:07:46.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-04-25T07:07:46.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0' '2') 2024-04-25T07:07:46.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T07:07:46.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T07:07:46.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T07:07:46.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/divergent-priors/1/type 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T07:07:46.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T07:07:46.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T07:07:46.837 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T07:07:46.837 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:07:46.837 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:07:46.838 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:46.838 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:46.838 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:07:46.838 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:07:46.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.25959/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T07:07:46.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T07:07:46.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T07:07:46.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T07:07:46.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T07:07:46.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:07:46.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:07:46.957 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:46.958 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:46.958 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:07:46.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:07:46.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T07:07:46.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok injectdataerr test existing_99 2024-04-25T07:07:47.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-04-25T07:07:47.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-04-25T07:07:57.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-04-25T07:07:57.088 INFO:tasks.workunit.client.0.smithi031.stderr:rados: no process found 2024-04-25T07:07:57.088 INFO:tasks.workunit.client.0.smithi031.stdout:killing all the osds 2024-04-25T07:07:57.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-04-25T07:07:57.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-04-25T07:07:57.507 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:57.507 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-04-25T07:07:47.236017+0000 29'101 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:58:14.500539+0000 0 0 2024-04-25T07:07:57.508 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:07:57.508 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:57.508 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:07:57.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:07:57.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:57.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:57.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:57.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:57.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:07:57.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:07:57.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:07:57.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-04-25T07:07:58.406 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:07:58.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:07:58.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-04-25T07:07:59.091 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:07:59.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:07:59.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-04-25T07:07:59.653 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:07:59.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T07:07:59.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-04-25T07:08:00.205 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T07:08:00.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:00.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-04-25T07:08:00.815 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already out. 2024-04-25T07:08:00.833 INFO:tasks.workunit.client.0.smithi031.stdout:bringing up non_divergent 0 2 2024-04-25T07:08:00.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-04-25T07:08:00.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-04-25T07:08:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:08:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-04-25T07:07:47.236017+0000 29'101 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:58:14.500539+0000 0 0 2024-04-25T07:08:01.241 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:08:01.241 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:08:01.242 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:08:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:08:01.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:08:01.263 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:08:01.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:08:01.310 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:01.309+0000 7f8eefdf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:01.313 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:01.309+0000 7f8eefdf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:01.314 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:01.309+0000 7f8eefdf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:01.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:08:01.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:01.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:08:01.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:01.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:01.671 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:08:01.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:01.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:01.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:01.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:01.861 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:01.857+0000 7f8eefdf77c0 -1 Falling back to public interface 2024-04-25T07:08:02.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:03.172 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:08:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:03.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:04.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:04.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:04.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:04.622 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:08:04.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:04.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:04.840 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:04.837+0000 7f8eefdf77c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:08:05.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:06.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:06.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:06.168 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:08:06.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:06.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:06.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:06.619 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/394387595,v1:127.0.0.1:6803/394387595] [v2:127.0.0.1:6804/394387595,v1:127.0.0.1:6805/394387595] exists,up d0a9b5c4-c2ec-465b-90bd-285d2f6545eb 2024-04-25T07:08:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-04-25T07:08:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:06.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:08:06.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:08:06.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:08:06.625 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:08:06.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:06.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:08:06.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:08:06.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:08:06.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:08:06.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:08:06.663 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:06.661+0000 7f64ccc3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:06.665 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:06.661+0000 7f64ccc3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:06.666 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:06.665+0000 7f64ccc3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:07.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:07.073 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:08:07.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:07.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:07.231 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:07.229+0000 7f64ccc3b7c0 -1 Falling back to public interface 2024-04-25T07:08:07.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:08.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:08.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:08.540 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:08:08.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:08.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:08.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:08.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:09.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:09.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:09.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:09.999 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:08:09.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:09.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:10.011 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:10.009+0000 7f64ccc3b7c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:08:10.566 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/3422715853,v1:127.0.0.1:6811/3422715853] [v2:127.0.0.1:6812/3422715853,v1:127.0.0.1:6813/3422715853] exists,up 4772361e-836b-4c5f-b574-d03f08754357 2024-04-25T07:08:10.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:10.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:10.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:10.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:10.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-04-25T07:08:11.174 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. 2024-04-25T07:08:11.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:11.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-04-25T07:08:11.776 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already in. 2024-04-25T07:08:11.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:08:11.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-04-25T07:08:11.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:08:11.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:08:11.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:08:11.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:08:11.793 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:08:11.794 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:08:11.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:08:11.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:08:11.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:08:11.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:08:11.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:08:11.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:08:11.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:08:11.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:08:11.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:08:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:08:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:08:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:08:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:08:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:08:12.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:08:12.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822660 2024-04-25T07:08:12.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822660 2024-04-25T07:08:12.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660' 2024-04-25T07:08:12.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:08:12.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:08:12.632 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:08:12.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:08:12.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:08:12.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:08:12.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:08:12.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:08:12.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T07:08:12.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T07:08:12.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-04-25T07:08:12.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:08:12.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822660 2024-04-25T07:08:12.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:08:12.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:08:12.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-04-25T07:08:12.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:08:12.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822660 2024-04-25T07:08:12.809 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 154618822660 2024-04-25T07:08:12.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-04-25T07:08:12.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:08:13.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822658 -lt 154618822660 2024-04-25T07:08:13.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:08:14.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:08:14.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:08:14.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822660 -lt 154618822660 2024-04-25T07:08:14.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:08:14.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757251 2024-04-25T07:08:14.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:08:14.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:08:14.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-04-25T07:08:14.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:08:14.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T07:08:14.786 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 163208757251 2024-04-25T07:08:14.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-04-25T07:08:14.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:08:15.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757251 -lt 163208757251 2024-04-25T07:08:15.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:08:15.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:15.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:08:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:15.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:15.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:15.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:15.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:15.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:15.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:16.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:16.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:16.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:16.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:16.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:16.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:08:16.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:08:16.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:08:16.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:16.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:08:16.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:16.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:16.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:16.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:16.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:16.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:16.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:16.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:17.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:17.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:17.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:17.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:17.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:17.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:17.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:08:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:08:18.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:18.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:18.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:18.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:18.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:18.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:18.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:18.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:19.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:19.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:19.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:19.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:19.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:19.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:20.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:20.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:20.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:08:20.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:20.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:08:20.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:20.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:20.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:20.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:20.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:20.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:20.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:20.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:21.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:21.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:21.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:21.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:21.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:21.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:22.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:22.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:22.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:08:22.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:22.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:08:23.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:23.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:23.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:23.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:23.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:23.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:23.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:23.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:23.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:23.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:23.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:24.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:24.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:24.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:24.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:24.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:24.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:08:24.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:24.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:08:26.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:26.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:26.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:26.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:26.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:26.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:26.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:26.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:26.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:26.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:26.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:26.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:27.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:27.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:27.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:27.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:27.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:27.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:27.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:27.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:27.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:27.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:27.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:27.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:08:27.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:27.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:08:31.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:31.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:31.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:31.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:31.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:31.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:31.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:31.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:31.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:31.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:31.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:31.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:32.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:32.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:32.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:32.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:32.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:32.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:08:32.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:32.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:08:39.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:39.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:39.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:39.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:39.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:39.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:39.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:39.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:39.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:39.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:39.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:39.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:40.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:40.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:40.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:40.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:40.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:08:40.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:40.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:08:48.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:48.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:48.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:48.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:48.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:48.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:48.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:48.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:48.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:48.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:48.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:48.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:49.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:49.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:49.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:49.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:49.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:08:49.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:08:50.272 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:08:50.272 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "11f96744-c610-4ff8-b0a3-3b86411e3bb0", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:08:50.272101+0000", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:08:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:08:50.274 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:08:50.275 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.276 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.277 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:05:27.745766Z", 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:05:27.745766Z", 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:08:50.278 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:08:50.279 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:08:50.280 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:08:50.281 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 41, 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:05:28.672762+0000", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:08:11.991529+0000", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:08:10.319060+0000", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:08:11.574661+0000", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:08:50.282 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 9, 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:08:50.283 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:07:12.135781+0000", 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:08:50.284 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 1, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 1, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 1, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 1, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 1, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:08:50.285 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "29", 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:08:50.286 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:08:50.287 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T07:08:50.288 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 2, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 2, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "d0a9b5c4-c2ec-465b-90bd-285d2f6545eb", 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 5, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 36, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 40, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 30, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.290 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.292 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:08:50.293 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 394387595 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/394387595", 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/394387595", 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/394387595", 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/394387595", 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:08:50.294 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "f0287046-382c-44ba-a3ab-51ca072bda43", 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:08:50.295 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 23, 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 31, 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.296 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.297 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.298 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 111237492 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.299 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/111237492", 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/111237492", 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/111237492", 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/111237492", 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "4772361e-836b-4c5f-b574-d03f08754357", 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:08:50.300 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 15, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 38, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 0, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 32, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.302 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:08:50.304 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3422715853 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3422715853", 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3422715853", 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3422715853", 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3422715853", 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:07:58.211843+0000", 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:06:01.348714+0000", 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.306 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:07:58.858234+0000", 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:06:35.100101+0000", 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:07:59.488484+0000", 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:08:50.307 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:07:07.607107+0000", 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:08:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:08:50.309 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/394387595,v1:127.0.0.1:6805/394387595]", 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:08:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:08:50.311 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:05:35.399531Z", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/394387595,v1:127.0.0.1:6803/394387595]", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/394387595,v1:127.0.0.1:6809/394387595]", 2024-04-25T07:08:50.312 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/394387595,v1:127.0.0.1:6807/394387595]", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/111237492,v1:127.0.0.1:6813/111237492]", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:08:50.314 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:06:09.027555Z", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:08:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/111237492,v1:127.0.0.1:6811/111237492]", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/111237492,v1:127.0.0.1:6817/111237492]", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/111237492,v1:127.0.0.1:6815/111237492]", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:08:50.316 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3422715853,v1:127.0.0.1:6813/3422715853]", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:08:50.317 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:08:50.318 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:06:41.658810Z", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:08:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3422715853,v1:127.0.0.1:6811/3422715853]", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3422715853,v1:127.0.0.1:6817/3422715853]", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3422715853,v1:127.0.0.1:6815/3422715853]", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:08:50.320 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:08:50.321 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 36 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 38 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 41, 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:08:50.322 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:08:50.324 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.325 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:08:50.326 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:08:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:08:50.328 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:08:50.329 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:08:50.331 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:08:50.332 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:08:50.333 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:08:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:08:50.335 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 0, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:08:50.336 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:08:50.337 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:08:50.338 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 1, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 61700, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 100, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 300, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 100, 2024-04-25T07:08:50.339 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 100, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 100, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 100, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:08:50.340 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:08:50.341 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:08:50.342 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 16, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 16, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 2, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 2, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:08:50.343 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83636, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2112, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489164, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036903936, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2162688, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 460116, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:08:50.344 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416932 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 1, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:08:50.345 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:08:50.346 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83636, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2112, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489164, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036903936, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2162688, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 460116, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:08:50.347 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416932 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 1, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:08:50.348 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 61700, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 100, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 300, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:08:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 100, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 100, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 100, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 100, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:08:50.350 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:08:50.351 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1228800, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 185100, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:08:50.352 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 16, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 16, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 2, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 2, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 154618822667 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.353 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672985 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 163208757258 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 1 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.354 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.355 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 5, 2024-04-25T07:08:50.356 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:08:02.297963+0000", 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 89, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 373, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 510, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:08:50.357 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:08:50.358 INFO:tasks.workunit.client.0.smithi031.stderr:report 228662867 2024-04-25T07:08:50.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:08:50.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-04-25T07:08:50.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-04-25T07:08:50.359 INFO:tasks.workunit.client.0.smithi031.stdout:writing non-divergent object existing_4 2024-04-25T07:08:50.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-04-25T07:08:50.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-04-25T07:08:50.694 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:08:50.694 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:08:12.185408+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:20:05.257612+0000 0 0 2024-04-25T07:08:50.694 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:08:50.694 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:08:50.694 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:08:50.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:08:50.822 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery 2024-04-25T07:08:50.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-04-25T07:08:50.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-04-25T07:08:51.241 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:08:51.241 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:08:12.185408+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:20:05.257612+0000 0 0 2024-04-25T07:08:51.241 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:08:51.241 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:08:51.241 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:08:51.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:51.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-04-25T07:08:51.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:08:51.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:08:51.260 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:08:51.260 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:51.260 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:08:51.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:08:51.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T07:08:51.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:08:51.375 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:08:51.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T07:08:51.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-04-25T07:08:51.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:08:51.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:08:51.389 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:08:51.389 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:51.389 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:08:51.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:08:51.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T07:08:51.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:08:51.492 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:08:51.503 INFO:tasks.workunit.client.0.smithi031.stdout:revive divergent 1 2024-04-25T07:08:51.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-04-25T07:08:51.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-04-25T07:08:51.920 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:08:51.920 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:08:12.185408+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:20:05.257612+0000 0 0 2024-04-25T07:08:51.920 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:08:51.920 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:08:51.920 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:08:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-04-25T07:08:52.630 INFO:tasks.workunit.client.0.smithi031.stderr:noup is set 2024-04-25T07:08:52.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T07:08:52.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:08:52.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:08:52.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:08:52.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:08:52.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:08:52.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:08:52.658 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:08:52.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:08:52.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:52.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:08:52.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:08:52.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:08:52.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:08:52.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:08:52.700 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:52.697+0000 7feca8b6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:52.702 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:52.701+0000 7feca8b6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:52.703 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:52.701+0000 7feca8b6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:53.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:08:53.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-04-25T07:08:53.249 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:53.245+0000 7feca8b6c7c0 -1 Falling back to public interface 2024-04-25T07:08:56.189 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:08:56.185+0000 7feca8b6c7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T07:08:58.101 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery divergent 2024-04-25T07:08:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-04-25T07:08:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-04-25T07:08:58.526 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:08:58.526 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:08:12.185408+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:20:05.257612+0000 0 0 2024-04-25T07:08:58.526 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:08:58.526 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:08:58.527 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:08:58.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-04-25T07:08:58.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:08:58.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:08:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:08:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:08:58.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:08:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-04-25T07:08:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:08:58.664 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:08:58.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-04-25T07:08:59.333 INFO:tasks.workunit.client.0.smithi031.stderr:noup is unset 2024-04-25T07:08:59.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-04-25T07:08:59.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:59.353 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:59.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:59.993 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/394387595,v1:127.0.0.1:6803/394387595] [v2:127.0.0.1:6804/394387595,v1:127.0.0.1:6805/394387595] exists,up d0a9b5c4-c2ec-465b-90bd-285d2f6545eb 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:08:59.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:59.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:59.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:59.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:59.995 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:08:59.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:59.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:00.578 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4255289577,v1:127.0.0.1:6819/4255289577] [v2:127.0.0.1:6820/4255289577,v1:127.0.0.1:6821/4255289577] exists,up f0287046-382c-44ba-a3ab-51ca072bda43 2024-04-25T07:09:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:00.579 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:09:00.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:00.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:00.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:01.025 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/3422715853,v1:127.0.0.1:6811/3422715853] [v2:127.0.0.1:6812/3422715853,v1:127.0.0.1:6813/3422715853] exists,up 4772361e-836b-4c5f-b574-d03f08754357 2024-04-25T07:09:01.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:01.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:01.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:01.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:01.460 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:01.460 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:08:12.185408+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:20:05.257612+0000 0 0 2024-04-25T07:09:01.460 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:09:01.460 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:01.460 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:09:01.476 INFO:tasks.workunit.client.0.smithi031.stdout:wait for peering 2024-04-25T07:09:01.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-04-25T07:09:01.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:01.890 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:01.890 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:08:12.185408+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:20:05.257612+0000 0 0 2024-04-25T07:09:01.890 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:09:01.890 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:01.891 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:09:01.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-04-25T07:09:02.053 INFO:tasks.workunit.client.0.smithi031.stdout:killing divergent 1 2024-04-25T07:09:02.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-04-25T07:09:02.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:02.463 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:02.463 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-04-25T07:09:00.075493+0000 41'101 47:139 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:09:02.463 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:09:02.463 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:02.463 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:09:02.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:09:02.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:09:02.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:09:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:09:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:09:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:09:02.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:09:02.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-04-25T07:09:02.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:02.587 INFO:tasks.workunit.client.0.smithi031.stdout:reviving divergent 1 2024-04-25T07:09:03.044 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:03.044 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-04-25T07:09:00.075493+0000 41'101 47:139 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:09:03.044 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:09:03.044 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:03.044 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:09:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:09:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:03.064 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:09:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:09:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:09:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:09:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:09:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:09:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:09:03.066 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:09:03.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:09:03.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:09:03.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:09:03.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:09:03.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:09:03.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:09:03.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:09:03.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:03.097+0000 7ff5f8fa07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:03.102 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:03.101+0000 7ff5f8fa07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:03.103 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:03.101+0000 7ff5f8fa07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:03.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:09:03.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:03.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:09:03.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:03.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:03.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:03.533 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:09:03.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:03.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:03.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:03.671 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:03.669+0000 7ff5f8fa07c0 -1 Falling back to public interface 2024-04-25T07:09:03.976 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4255289577,v1:127.0.0.1:6819/4255289577] [v2:127.0.0.1:6820/4255289577,v1:127.0.0.1:6821/4255289577] exists,up f0287046-382c-44ba-a3ab-51ca072bda43 2024-04-25T07:09:03.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:03.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:03.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:03.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-04-25T07:09:06.352 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:06.349+0000 7ff5f8fa07c0 -1 osd.1 47 log_to_monitors true 2024-04-25T07:09:23.978 INFO:tasks.workunit.client.0.smithi031.stdout:allowing recovery 2024-04-25T07:09:23.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-04-25T07:09:23.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T07:09:07.734942+0000 47'102 51:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:44:59.841451+0000 0 0 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:09:24.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:09:24.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:09:24.563 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:09:24.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:09:24.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:09:24.732 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:09:24.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:09:24.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:09:24.898 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:09:24.913 INFO:tasks.workunit.client.0.smithi031.stdout:reading divergent objects 2024-04-25T07:09:24.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-04-25T07:09:24.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:25.322 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:25.323 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T07:09:07.734942+0000 47'102 51:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:12.164834+0000 0'0 2024-04-25T07:07:12.164834+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:44:59.841451+0000 0 0 2024-04-25T07:09:25.323 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:09:25.323 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:25.323 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:09:25.341 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-04-25T07:09:25.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-04-25T07:09:25.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:09:25.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:09:25.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:09:25.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:09:25.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:09:25.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:09:25.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:09:25.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:09:25.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-04-25T07:09:25.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T07:09:25.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4614.0:1 2024-04-25T07:07:46.419481+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4614.0:1 2024-04-25T07:07:46.419481+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-04-25T07:09:25.504 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4614.0:1 2024-04-25T07:07:46.419481+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:09:25.505 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:08:59.973+0000 7fec8caf6640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:09:25.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T07:09:25.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:09:25.509 INFO:tasks.workunit.client.0.smithi031.stdout:success 2024-04-25T07:09:25.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-04-25T07:09:25.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-04-25T07:09:25.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:09:25.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:09:26.117 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T07:09:26.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-04-25T07:09:26.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:09:26.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:09:26.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:09:26.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:09:26.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:09:26.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:09:26.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:09:26.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:09:26.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:09:26.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:09:26.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:09:26.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:09:26.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:09:26.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:09:26.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:09:26.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:09:26.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:09:26.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:09:26.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:09:26.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:09:26.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:09:26.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:09:26.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:09:26.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:09:26.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:09:26.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:09:26.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:09:26.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:09:26.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:09:26.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:09:26.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:26.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:26.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:09:26.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:09:26.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:09:26.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:09:26.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:09:26.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:09:26.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:09:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:09:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:09:26.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:09:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:09:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:09:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:09:26.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:09:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:09:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:09:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:09:26.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:09:26.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:26.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:09:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:09:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:09:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:09:26.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:09:26.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:26.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:26.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.25959 2024-04-25T07:09:26.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:09:26.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:09:26.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:09:26.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:09:26.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-04-25T07:09:26.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-04-25T07:09:26.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-04-25T07:09:26.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-04-25T07:09:26.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-04-25T07:09:26.498 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-04-25T07:09:26.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-04-25T07:09:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-04-25T07:09:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:09:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:09:26.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:09:26.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:09:26.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:26.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:26.943 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:26.943 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:26.943 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:26.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:09:26.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:09:27.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:09:27.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:09:27.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:09:27.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:09:27.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:27.165 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:27.166 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:09:27.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:09:27.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get fsid 2024-04-25T07:09:27.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:09:27.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:09:27.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:09:27.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:09:27.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:09:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:09:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:09:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:09:27.285 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:27.285 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:27.285 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:27.285 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:09:27.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:09:27.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get mon_host 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:09:27.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:09:28.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:09:28.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:28.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:28.318 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:28.318 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:28.318 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:28.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:09:28.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:09:28.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:09:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:09:28.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:09:28.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:09:28.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:09:28.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a136043e-f78a-4630-a55a-6bb7d421c30d 2024-04-25T07:09:28.362 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 a136043e-f78a-4630-a55a-6bb7d421c30d 2024-04-25T07:09:28.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a136043e-f78a-4630-a55a-6bb7d421c30d' 2024-04-25T07:09:28.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:09:28.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoASpmUQgRFxAAVdPcLbA6cL/pXyhTDuVz6A== 2024-04-25T07:09:28.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoASpmUQgRFxAAVdPcLbA6cL/pXyhTDuVz6A=="}' 2024-04-25T07:09:28.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a136043e-f78a-4630-a55a-6bb7d421c30d -i td/divergent-priors/0/new.json 2024-04-25T07:09:28.674 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:09:28.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:09:28.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoASpmUQgRFxAAVdPcLbA6cL/pXyhTDuVz6A== --osd-uuid a136043e-f78a-4630-a55a-6bb7d421c30d 2024-04-25T07:09:28.742 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.737+0000 7f852e6467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:28.744 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.741+0000 7f852e6467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:28.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.741+0000 7f852e6467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:28.746 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.745+0000 7f852e6467c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:09:28.746 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.745+0000 7f852e6467c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:09:28.746 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.745+0000 7f852e6467c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:09:28.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:28.781+0000 7f852e6467c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:09:34.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:09:34.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:09:34.902 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:09:34.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:09:34.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:09:35.494 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:09:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:09:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:09:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:09:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:09:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:09:35.536 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:35.533+0000 7fcc4546d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:35.538 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:35.537+0000 7fcc4546d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:35.539 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:35.537+0000 7fcc4546d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:35.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:09:35.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:35.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:35.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:36.111 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:36.109+0000 7fcc4546d7c0 -1 Falling back to public interface 2024-04-25T07:09:36.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:37.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:37.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:37.345 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:09:37.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:09:37.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:37.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:37.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:38.670 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:09:38.668+0000 7fcc4546d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:09:38.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:38.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:38.801 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:09:38.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:09:38.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:38.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:39.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:40.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:40.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:40.276 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:09:40.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:09:40.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:40.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:40.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:41.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:41.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:41.762 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:09:41.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:09:41.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:41.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:42.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:43.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:43.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:43.239 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:09:43.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:09:43.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:43.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:43.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:44.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:44.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:44.771 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:09:44.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:09:44.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:44.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:45.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:46.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:46.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:46.207 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:09:46.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:09:46.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:46.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:46.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:47.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:47.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:47.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:09:47.788 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:09:47.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:47.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:48.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:49.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:49.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:49.235 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:09:49.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:09:49.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:49.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:49.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:50.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:50.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:50.739 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:09:50.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:09:50.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:50.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:51.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:52.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:52.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:52.352 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:09:52.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:09:52.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:52.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:53.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:54.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:54.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:54.325 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:09:54.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:09:54.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:54.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:55.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:56.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:56.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:56.047 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:09:56.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:09:56.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:56.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:57.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:58.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:58.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:58.086 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:09:58.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:09:58.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:58.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:59.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:00.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:00.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:00.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:10:00.136 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:10:00.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:00.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:00.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:01.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:01.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:01.694 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:10:01.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:10:01.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:01.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:02.128 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2017587385,v1:127.0.0.1:6803/2017587385] [v2:127.0.0.1:6804/2017587385,v1:127.0.0.1:6805/2017587385] exists,up a136043e-f78a-4630-a55a-6bb7d421c30d 2024-04-25T07:10:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:10:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:10:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:10:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:10:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:10:02.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:10:02.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:10:02.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:10:02.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:10:02.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:10:02.136 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:10:02.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9bf21071-871f-48fe-b320-e7acf88628d0' 2024-04-25T07:10:02.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:10:02.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKASpmr3H5CRAAFKAgwnXiTgbPHkI5NAIztg== 2024-04-25T07:10:02.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKASpmr3H5CRAAFKAgwnXiTgbPHkI5NAIztg=="}' 2024-04-25T07:10:02.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9bf21071-871f-48fe-b320-e7acf88628d0 -i td/divergent-priors/1/new.json 2024-04-25T07:10:02.689 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:10:02.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:10:02.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKASpmr3H5CRAAFKAgwnXiTgbPHkI5NAIztg== --osd-uuid 9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:10:02.751 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.748+0000 7fb16f44c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:02.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.748+0000 7fb16f44c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:02.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.752+0000 7fb16f44c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:02.755 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.752+0000 7fb16f44c7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:10:02.755 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.752+0000 7fb16f44c7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:10:02.755 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.752+0000 7fb16f44c7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:10:02.794 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:02.792+0000 7fb16f44c7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:10:08.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:10:08.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:10:08.649 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:10:08.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:10:08.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:10:09.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:10:09.236 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:10:09.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:09.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:10:09.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:10:09.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:10:09.277 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:09.272+0000 7f1dd0fe67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:09.295 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:09.292+0000 7f1dd0fe67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:09.296 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:09.292+0000 7f1dd0fe67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:09.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:10:09.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:09.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:10:09.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:09.750 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:10:09.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:09.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:09.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:09.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:09.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:09.883 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:09.880+0000 7f1dd0fe67c0 -1 Falling back to public interface 2024-04-25T07:10:10.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:11.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:11.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:11.234 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:10:11.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:11.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:11.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:11.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:12.552 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:12.548+0000 7f1dd0fe67c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:10:12.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:12.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:12.671 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:10:12.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:12.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:12.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:13.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:10:14.246 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:10:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:14.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:15.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:15.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:15.705 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:10:15.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:10:15.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:15.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:16.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:17.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:17.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:17.370 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:10:17.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:10:17.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:17.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:17.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:18.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:18.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:18.882 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:10:18.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:10:18.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:18.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:19.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:20.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:20.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:20.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:10:20.447 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:10:20.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:20.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:20.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:21.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:21.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:21.897 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:10:21.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:10:21.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:21.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:22.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:23.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:23.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:23.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:10:23.472 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:10:23.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:23.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:23.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:24.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:24.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:24.916 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:10:24.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:10:24.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:24.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:25.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:26.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:26.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:26.360 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:10:26.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:10:26.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:26.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:27.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:28.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:28.272 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:10:28.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:10:28.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:28.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:29.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:30.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:30.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:30.143 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:10:30.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:10:30.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:30.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:30.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:31.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:31.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:31.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:10:31.953 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:10:31.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:31.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:32.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:33.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:33.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:33.877 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:10:33.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:10:33.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:33.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:34.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:35.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:35.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:35.635 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:10:35.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:10:35.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:35.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:10:36.136 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3314957454,v1:127.0.0.1:6811/3314957454] [v2:127.0.0.1:6812/3314957454,v1:127.0.0.1:6813/3314957454] exists,up 9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:10:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:36.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:36.140 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:36.140 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:36.140 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:10:36.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:10:36.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:10:36.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:36.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:10:36.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:10:36.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:36.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a0ee02d-b1be-478c-958f-d830803e2e77 2024-04-25T07:10:36.142 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 6a0ee02d-b1be-478c-958f-d830803e2e77 2024-04-25T07:10:36.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6a0ee02d-b1be-478c-958f-d830803e2e77' 2024-04-25T07:10:36.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:10:36.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsASpmI7/jCRAA+jt0hzl6u3nsfMJwGJ730g== 2024-04-25T07:10:36.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsASpmI7/jCRAA+jt0hzl6u3nsfMJwGJ730g=="}' 2024-04-25T07:10:36.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a0ee02d-b1be-478c-958f-d830803e2e77 -i td/divergent-priors/2/new.json 2024-04-25T07:10:36.732 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:10:36.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:10:36.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsASpmI7/jCRAA+jt0hzl6u3nsfMJwGJ730g== --osd-uuid 6a0ee02d-b1be-478c-958f-d830803e2e77 2024-04-25T07:10:36.792 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.792+0000 7f91cc4297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:36.794 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.796+0000 7f91cc4297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:36.796 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.796+0000 7f91cc4297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:36.796 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.796+0000 7f91cc4297c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:10:36.796 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.796+0000 7f91cc4297c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:10:36.797 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.796+0000 7f91cc4297c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:10:36.853 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:36.856+0000 7f91cc4297c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:10:42.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:10:42.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:10:42.885 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:10:42.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:10:42.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:10:43.479 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:10:43.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:10:43.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:43.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:10:43.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:10:43.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:10:43.520 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:43.520+0000 7f4749d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:43.543 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:43.544+0000 7f4749d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:43.545 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:43.544+0000 7f4749d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:43.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:10:43.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:43.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:10:43.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:43.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:43.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:43.975 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:10:43.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:43.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:43.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:44.110 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:44.112+0000 7f4749d2b7c0 -1 Falling back to public interface 2024-04-25T07:10:44.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:45.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:45.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:45.459 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:10:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:45.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:46.528 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:10:46.528+0000 7f4749d2b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:10:46.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:46.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:46.954 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:10:46.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:46.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:46.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:47.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:48.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:48.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:48.685 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:10:48.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:10:48.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:48.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:49.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:50.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:50.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:50.131 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:10:50.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:10:50.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:50.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:50.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:51.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:51.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:51.675 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:10:51.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:10:51.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:51.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:52.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:53.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:53.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:53.151 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:10:53.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:10:53.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:53.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:53.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:54.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:54.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:54.686 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:10:54.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:10:54.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:54.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:56.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:56.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:56.136 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:10:56.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:10:56.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:56.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:56.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:57.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:57.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:57.652 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:10:57.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:10:57.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:57.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:59.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:59.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:59.111 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:10:59.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:10:59.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:59.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:01.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:01.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:01.075 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:11:01.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:11:01.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:01.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:01.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:02.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:02.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:02.807 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:11:02.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:11:02.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:02.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:03.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:04.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:04.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:04.337 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:11:04.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:11:04.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:04.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:04.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:05.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:05.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:05.925 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:11:05.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:11:05.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:05.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:06.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:07.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:07.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:07.506 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:11:07.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:11:07.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:07.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:08.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:09.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:09.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:09.777 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:11:09.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:11:09.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:09.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:10.240 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4239561798,v1:127.0.0.1:6819/4239561798] [v2:127.0.0.1:6820/4239561798,v1:127.0.0.1:6821/4239561798] exists,up 6a0ee02d-b1be-478c-958f-d830803e2e77 2024-04-25T07:11:10.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:10.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:10.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:10.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-04-25T07:11:11.008 INFO:tasks.workunit.client.0.smithi031.stderr:noout is set 2024-04-25T07:11:11.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-04-25T07:11:11.878 INFO:tasks.workunit.client.0.smithi031.stderr:noin is set 2024-04-25T07:11:11.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-04-25T07:11:12.573 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T07:11:12.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-04-25T07:11:12.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:11:13.226 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T07:11:13.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:11:14.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-04-25T07:11:14.918 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 3 2024-04-25T07:11:14.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-04-25T07:11:15.605 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 2 2024-04-25T07:11:15.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-04-25T07:11:15.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:11:15.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:11:16.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:11:16.101 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:11:16.101 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:11:16.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:11:16.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:16.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:11:16.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T07:11:16.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T07:11:16.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T07:11:16.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:16.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:11:16.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:11:16.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:11:16.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-04-25T07:11:16.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:16.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:11:16.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-04-25T07:11:16.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-04-25T07:11:16.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509448' 2024-04-25T07:11:16.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:16.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T07:11:16.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:16.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:11:16.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T07:11:16.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:16.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T07:11:16.619 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836502 2024-04-25T07:11:16.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T07:11:16.620 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:17.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836502 2024-04-25T07:11:17.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:11:18.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:11:18.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:18.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-25T07:11:18.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:18.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:11:18.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:18.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:11:18.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:18.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:11:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:11:18.500 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:11:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:11:18.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:11:19.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-25T07:11:19.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:19.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-04-25T07:11:19.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:19.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:11:19.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-04-25T07:11:19.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:19.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-04-25T07:11:19.013 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509448 2024-04-25T07:11:19.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-25T07:11:19.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:11:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509448 2024-04-25T07:11:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-04-25T07:11:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:11:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:11:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:11:19.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:11:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:11:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:11:19.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:11:19.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:11:19.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:11:19.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:11:19.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:11:19.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:11:19.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:11:19.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:11:19.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:11:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:11:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:11:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:11:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:11:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:20.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:11:20.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T07:11:20.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T07:11:20.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T07:11:20.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:20.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:11:20.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T07:11:20.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T07:11:20.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2024-04-25T07:11:20.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:20.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:11:20.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T07:11:20.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T07:11:20.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509451' 2024-04-25T07:11:20.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:20.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T07:11:20.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:20.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:11:20.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T07:11:20.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:20.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T07:11:20.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T07:11:20.522 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836504 2024-04-25T07:11:20.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:21.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836504 2024-04-25T07:11:21.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:11:22.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:11:22.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:22.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836504 2024-04-25T07:11:22.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:22.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T07:11:22.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:22.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:11:22.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T07:11:22.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:22.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T07:11:22.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T07:11:22.444 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672977 2024-04-25T07:11:22.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:11:22.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672977 2024-04-25T07:11:22.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:22.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T07:11:22.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:22.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:11:22.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T07:11:22.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:22.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T07:11:22.911 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509451 2024-04-25T07:11:22.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T07:11:22.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:11:23.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-25T07:11:23.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:11:23.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:23.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:23.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:11:23.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:23.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:23.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:23.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:23.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:23.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:23.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:11:24.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:24.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:24.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:24.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:11:24.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:11:24.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:11:24.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-04-25T07:11:24.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-04-25T07:11:25.395 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:11:25.411 INFO:tasks.workunit.client.0.smithi031.stdout:primary and soon to be divergent is 1 2024-04-25T07:11:25.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-04-25T07:11:25.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-04-25T07:11:25.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:11:25.812 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:25.812 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:11:14.135150+0000 0'0 29:21 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:38:02.584931+0000 0 0 2024-04-25T07:11:25.812 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:11:25.812 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:25.812 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:11:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-04-25T07:11:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:11:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-04-25T07:11:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-04-25T07:11:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:11:25.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-04-25T07:11:25.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-04-25T07:11:25.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:11:25.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-04-25T07:11:25.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-04-25T07:11:25.831 INFO:tasks.workunit.client.0.smithi031.stdout:writing initial objects 2024-04-25T07:11:25.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-04-25T07:11:25.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-04-25T07:11:25.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:25.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-04-25T07:11:25.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:25.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-04-25T07:11:26.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:26.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-04-25T07:11:26.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:26.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-04-25T07:11:26.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:26.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-04-25T07:11:26.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:26.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-04-25T07:11:26.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:26.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-04-25T07:11:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-04-25T07:11:27.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-04-25T07:11:27.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-04-25T07:11:27.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-04-25T07:11:27.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-04-25T07:11:27.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-04-25T07:11:27.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-04-25T07:11:27.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:27.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-04-25T07:11:28.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:28.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-04-25T07:11:28.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:28.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-04-25T07:11:28.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:28.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-04-25T07:11:28.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:28.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-04-25T07:11:28.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:28.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-04-25T07:11:28.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:28.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-04-25T07:11:29.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:29.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-04-25T07:11:29.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:29.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-04-25T07:11:29.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:29.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-04-25T07:11:29.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:29.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-04-25T07:11:29.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:29.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-04-25T07:11:29.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:29.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-04-25T07:11:30.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-04-25T07:11:30.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-04-25T07:11:30.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-04-25T07:11:30.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-04-25T07:11:30.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-04-25T07:11:30.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-04-25T07:11:30.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:30.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-04-25T07:11:31.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-04-25T07:11:31.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-04-25T07:11:31.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-04-25T07:11:31.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-04-25T07:11:31.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-04-25T07:11:31.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-04-25T07:11:31.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:31.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-04-25T07:11:32.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:32.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-04-25T07:11:32.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:32.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-04-25T07:11:32.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:32.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-04-25T07:11:32.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:32.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-04-25T07:11:32.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:32.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-04-25T07:11:32.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:32.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-04-25T07:11:33.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-04-25T07:11:33.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-04-25T07:11:33.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-04-25T07:11:33.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-04-25T07:11:33.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-04-25T07:11:33.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-04-25T07:11:33.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:33.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-04-25T07:11:34.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-04-25T07:11:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-04-25T07:11:34.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-04-25T07:11:34.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-04-25T07:11:34.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-04-25T07:11:34.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-04-25T07:11:34.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:34.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-04-25T07:11:35.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:35.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-04-25T07:11:35.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:35.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-04-25T07:11:35.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:35.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-04-25T07:11:35.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:35.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-04-25T07:11:35.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:35.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-04-25T07:11:35.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:35.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-04-25T07:11:36.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:36.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-04-25T07:11:36.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:36.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-04-25T07:11:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-04-25T07:11:36.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:36.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-04-25T07:11:36.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:36.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-04-25T07:11:36.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:36.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-04-25T07:11:37.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-04-25T07:11:37.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-04-25T07:11:37.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-04-25T07:11:37.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-04-25T07:11:37.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-04-25T07:11:37.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-04-25T07:11:37.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:37.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-04-25T07:11:38.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:38.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-04-25T07:11:38.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:38.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-04-25T07:11:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-04-25T07:11:38.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:38.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-04-25T07:11:38.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:38.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-04-25T07:11:38.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:38.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-04-25T07:11:39.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-04-25T07:11:39.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-04-25T07:11:39.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-04-25T07:11:39.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-04-25T07:11:39.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-04-25T07:11:39.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-04-25T07:11:39.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:39.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-04-25T07:11:40.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-04-25T07:11:40.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-04-25T07:11:40.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-04-25T07:11:40.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-04-25T07:11:40.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-04-25T07:11:40.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-04-25T07:11:40.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:11:40.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-04-25T07:11:41.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:11:41.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-04-25T07:11:41.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:11:41.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:11:41.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:11:41.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:11:41.116 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:11:41.116 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:11:41.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:11:41.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:11:41.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:11:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:11:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:11:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:11:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:11:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:11:41.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:11:41.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:11:41.718 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:11:41.718 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:11:41.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:11:41.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:41.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:11:41.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T07:11:41.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T07:11:41.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T07:11:41.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:41.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:11:42.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672984 2024-04-25T07:11:42.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672984 2024-04-25T07:11:42.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984' 2024-04-25T07:11:42.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:42.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:11:42.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-04-25T07:11:42.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-04-25T07:11:42.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984 2-64424509457' 2024-04-25T07:11:42.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:42.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T07:11:42.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:42.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:11:42.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T07:11:42.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:42.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T07:11:42.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T07:11:42.233 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836511 2024-04-25T07:11:42.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:42.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T07:11:42.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:11:43.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:11:43.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:44.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T07:11:44.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:44.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672984 2024-04-25T07:11:44.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:44.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:11:44.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672984 2024-04-25T07:11:44.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:44.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672984 2024-04-25T07:11:44.133 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672984 2024-04-25T07:11:44.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-25T07:11:44.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:11:44.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-25T07:11:44.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:44.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-04-25T07:11:44.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:44.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:11:44.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-04-25T07:11:44.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:44.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-04-25T07:11:44.826 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509457 2024-04-25T07:11:44.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-25T07:11:44.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:11:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-04-25T07:11:45.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:11:45.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:45.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:45.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:46.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:11:46.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:46.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:46.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:46.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:11:46.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:11:46.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:11:46.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-04-25T07:11:46.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:11:46.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:11:46.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:11:46.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:11:47.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-04-25T07:11:47.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-04-25T07:11:47.436 INFO:tasks.workunit.client.0.smithi031.stdout:blackholing osds 0 2 2024-04-25T07:11:47.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:11:47.852 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:47.852 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:11:14.135150+0000 29'100 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:38:02.584931+0000 0 0 2024-04-25T07:11:47.852 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:11:47.852 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:47.852 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:11:47.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:11:47.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-04-25T07:11:47.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:11:47.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:11:47.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:47.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:47.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:11:47.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:11:47.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:11:47.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:11:47.977 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:11:47.977 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:11:47.977 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:11:47.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:11:47.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-04-25T07:11:47.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:11:47.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:11:47.992 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:47.992 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:47.992 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:11:47.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:11:47.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:11:47.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:11:48.106 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:11:48.106 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:11:48.106 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:11:48.119 INFO:tasks.workunit.client.0.smithi031.stdout:create new divergent objects 2024-04-25T07:11:48.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-04-25T07:11:48.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-04-25T07:11:48.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:11:48.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-04-25T07:11:48.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:11:48.122 INFO:tasks.workunit.client.0.smithi031.stdout:writing divergent objects 2024-04-25T07:11:48.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-04-25T07:11:48.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-04-25T07:11:48.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-04-25T07:11:48.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:11:48.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:11:48.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:11:48.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-04-25T07:11:48.125 INFO:tasks.workunit.client.0.smithi031.stdout:remove divergent objects 2024-04-25T07:11:48.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:11:48.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-04-25T07:11:48.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:11:48.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-04-25T07:11:48.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-04-25T07:11:48.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:11:48.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-04-25T07:11:48.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-04-25T07:11:48.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-04-25T07:11:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-04-25T07:11:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-04-25T07:11:58.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-04-25T07:11:58.138 INFO:tasks.workunit.client.0.smithi031.stderr:rados: no process found 2024-04-25T07:11:58.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-04-25T07:11:58.138 INFO:tasks.workunit.client.0.smithi031.stdout:killing all the osds 2024-04-25T07:11:58.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:11:58.561 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:58.561 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-04-25T07:11:14.135150+0000 29'106 29:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:38:02.584931+0000 0 0 2024-04-25T07:11:58.561 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:11:58.561 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:58.561 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:11:58.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:11:58.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:11:58.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:11:58.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:11:58.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:11:58.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:11:58.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:11:58.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:11:58.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-04-25T07:11:59.221 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:11:59.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:11:59.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-04-25T07:11:59.912 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:11:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:11:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-04-25T07:12:00.492 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:12:00.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:12:00.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-04-25T07:12:01.034 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T07:12:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:12:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-04-25T07:12:01.594 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already out. 2024-04-25T07:12:01.612 INFO:tasks.workunit.client.0.smithi031.stdout:bringing up non_divergent 0 2 2024-04-25T07:12:01.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-04-25T07:12:01.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:12:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:12:02.035 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-04-25T07:11:14.135150+0000 29'106 29:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:38:02.584931+0000 0 0 2024-04-25T07:12:02.036 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:12:02.036 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:12:02.036 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:12:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:12:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-04-25T07:12:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:02.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:12:02.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:12:02.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:12:02.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:02.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:02.059 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:02.060 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:02.060 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:12:02.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:12:02.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:12:02.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:02.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:12:02.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:12:02.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:12:02.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:12:02.062 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:12:02.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:02.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:12:02.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:12:02.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:12:02.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:12:02.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:12:02.092 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:02.091+0000 7fb7d6fea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:02.094 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:02.095+0000 7fb7d6fea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:02.095 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:02.095+0000 7fb7d6fea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:02.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:12:02.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:02.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:12:02.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:02.531 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:12:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:02.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:02.643 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:02.643+0000 7fb7d6fea7c0 -1 Falling back to public interface 2024-04-25T07:12:02.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:03.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:03.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:03.967 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:12:03.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:12:03.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:03.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:04.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:05.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:05.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:05.418 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:12:05.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:12:05.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:05.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:05.601 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:05.603+0000 7fb7d6fea7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:12:05.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:06.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:06.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:06.931 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:12:06.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:12:06.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:06.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:07.415 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/4293982427,v1:127.0.0.1:6803/4293982427] [v2:127.0.0.1:6804/4293982427,v1:127.0.0.1:6805/4293982427] exists,up a136043e-f78a-4630-a55a-6bb7d421c30d 2024-04-25T07:12:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:12:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:12:07.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:12:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:12:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:12:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:12:07.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:12:07.422 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:12:07.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:12:07.468 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:07.467+0000 7fe135a077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:07.496 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:07.495+0000 7fe135a077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:07.497 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:07.495+0000 7fe135a077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:07.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:12:07.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:07.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:12:07.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:07.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:07.870 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:12:07.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:07.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:07.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:07.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:12:08.064 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:08.063+0000 7fe135a077c0 -1 Falling back to public interface 2024-04-25T07:12:08.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:09.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:09.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:09.371 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:12:09.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:12:09.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:09.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:12:09.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:10.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:10.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:10.839 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:12:10.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:12:10.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:10.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:12:10.905 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:12:10.907+0000 7fe135a077c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:12:11.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:12.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:12.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:12.317 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:12:12.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:12:12.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:12.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:12:12.801 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2388680616,v1:127.0.0.1:6811/2388680616] [v2:127.0.0.1:6812/2388680616,v1:127.0.0.1:6813/2388680616] exists,up 6a0ee02d-b1be-478c-958f-d830803e2e77 2024-04-25T07:12:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:12:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-04-25T07:12:13.416 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. 2024-04-25T07:12:13.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:12:13.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-04-25T07:12:14.249 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already in. 2024-04-25T07:12:14.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:12:14.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-04-25T07:12:14.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:12:14.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:12:14.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:12:14.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:12:14.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:12:14.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:12:14.271 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:12:14.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:12:14.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:12:14.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:12:14.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:12:14.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:12:14.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:12:14.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:12:14.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:12:14.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:12:14.828 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:12:14.828 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:12:14.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:12:14.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:14.829 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:12:15.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822660 2024-04-25T07:12:15.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822660 2024-04-25T07:12:15.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660' 2024-04-25T07:12:15.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:15.004 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:12:15.137 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:12:15.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:12:15.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:12:15.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:12:15.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:15.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:12:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T07:12:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T07:12:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-04-25T07:12:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:15.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:15.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822660 2024-04-25T07:12:15.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:12:15.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-04-25T07:12:15.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:15.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822660 2024-04-25T07:12:15.419 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 154618822660 2024-04-25T07:12:15.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-04-25T07:12:15.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:12:15.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822660 -lt 154618822660 2024-04-25T07:12:15.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:15.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757251 2024-04-25T07:12:15.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:15.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:12:15.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-04-25T07:12:15.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:15.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T07:12:15.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-04-25T07:12:15.919 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 163208757251 2024-04-25T07:12:15.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:12:16.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757249 -lt 163208757251 2024-04-25T07:12:16.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:12:17.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:12:17.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:12:17.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757252 -lt 163208757251 2024-04-25T07:12:17.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:12:17.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:17.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:18.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:12:18.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:18.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:18.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:18.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:18.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:18.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:18.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:18.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:18.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:18.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:18.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:12:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:12:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:12:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:12:19.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:19.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:19.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:19.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:19.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:19.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:19.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:19.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:19.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:19.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:19.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:19.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:20.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:20.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:21.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:21.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:21.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:12:21.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:21.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:12:21.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:21.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:21.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:21.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:21.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:21.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:21.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:21.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:21.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:21.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:21.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:21.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:22.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:22.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:22.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:22.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:22.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:22.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:22.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:22.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:22.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:22.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:22.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:22.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:12:22.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:22.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:12:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:23.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:23.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:23.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:23.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:23.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:23.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:23.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:23.699 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:23.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:23.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:24.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:24.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:24.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:24.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:24.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:24.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:24.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:24.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:24.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:24.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:24.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:24.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:12:24.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:24.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:12:25.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:25.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:25.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:25.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:25.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:25.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:25.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:25.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:26.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:26.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:26.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:26.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:26.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:26.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:26.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:12:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:12:28.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:28.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:28.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:28.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:28.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:28.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:28.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:28.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:29.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:29.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:29.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:29.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:29.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:29.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:29.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:29.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:29.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:29.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:29.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:29.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:29.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:30.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:30.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:30.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:12:30.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:30.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:12:33.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:33.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:33.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:33.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:33.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:33.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:33.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:33.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:34.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:34.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:34.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:34.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:34.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:34.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:34.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:12:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:12:41.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:41.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:41.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:41.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:41.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:41.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:41.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:41.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:42.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:42.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:42.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:42.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:42.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:42.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:42.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:42.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:42.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:42.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:42.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:42.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:42.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:43.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:43.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:43.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:12:43.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:43.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:12:50.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:50.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:50.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:50.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:50.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:50.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:50.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:50.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:51.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:51.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:51.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:51.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:51.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:51.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:51.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:52.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:52.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:52.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:12:52.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "f951f6d5-344d-487b-9fd6-0ae729298bf5", 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:12:52.784945+0000", 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:12:52.785 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:12:52.786 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:12:52.787 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:12:52.788 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.789 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:09:26.528021Z", 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:09:26.528021Z", 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:12:52.790 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:12:52.791 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.792 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:12:52.793 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 42, 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:09:27.464635+0000", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:12:14.138917+0000", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:12:11.248628+0000", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:12:13.909086+0000", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:12:52.794 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 9, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:12:52.795 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:11:12.938535+0000", 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:12:52.796 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 1, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 1, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 1, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 1, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 1, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:12:52.797 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "29", 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:12:52.798 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:12:52.799 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:12:52.800 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 2, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 2, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:12:52.801 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "a136043e-f78a-4630-a55a-6bb7d421c30d", 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 5, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 36, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 41, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 30, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:12:52.802 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.803 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:12:52.804 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.805 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/4293982427", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/4293982427", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/4293982427", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/4293982427", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:12:52.806 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "9bf21071-871f-48fe-b320-e7acf88628d0", 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 23, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 31, 2024-04-25T07:12:52.807 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.808 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.809 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.810 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3314957454", 2024-04-25T07:12:52.811 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3314957454", 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3314957454", 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3314957454", 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "6a0ee02d-b1be-478c-958f-d830803e2e77", 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 15, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:12:52.812 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 38, 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 0, 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 32, 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.813 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.814 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.815 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.816 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/2388680616", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/2388680616", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2388680616", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2388680616", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:12:52.817 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:11:59.049882+0000", 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:10:00.252359+0000", 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.818 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:11:59.663915+0000", 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:10:35.072730+0000", 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:12:00.318626+0000", 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:12:52.819 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:11:08.853092+0000", 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:12:52.820 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:12:52.821 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/4293982427,v1:127.0.0.1:6805/4293982427]", 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:12:52.822 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:12:52.823 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:09:34.207300Z", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/4293982427,v1:127.0.0.1:6803/4293982427]", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/4293982427,v1:127.0.0.1:6809/4293982427]", 2024-04-25T07:12:52.824 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/4293982427,v1:127.0.0.1:6807/4293982427]", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:12:52.825 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3314957454,v1:127.0.0.1:6813/3314957454]", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:12:52.826 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:10:07.981880Z", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:12:52.827 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3314957454,v1:127.0.0.1:6811/3314957454]", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3314957454,v1:127.0.0.1:6817/3314957454]", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3314957454,v1:127.0.0.1:6815/3314957454]", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:12:52.828 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/2388680616,v1:127.0.0.1:6813/2388680616]", 2024-04-25T07:12:52.829 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:12:52.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:10:42.218183Z", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:12:52.831 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/2388680616,v1:127.0.0.1:6811/2388680616]", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2388680616,v1:127.0.0.1:6817/2388680616]", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2388680616,v1:127.0.0.1:6815/2388680616]", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:12:52.832 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 36 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.833 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 38 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 42, 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.834 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.835 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.836 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:12:52.837 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.838 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.839 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:12:52.840 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:12:52.841 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.842 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:12:52.843 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:12:52.844 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:12:52.846 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 0, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:12:52.847 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:12:52.848 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:12:52.848 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:12:52.848 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:12:52.848 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:12:52.848 INFO:tasks.workunit.client.0.smithi031.stderr:report 2271119742 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.849 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:12:52.850 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 1, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 61700, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 100, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 300, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 100, 2024-04-25T07:12:52.851 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 100, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 100, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 100, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:12:52.852 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:12:52.853 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:12:52.854 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 16, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 16, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 2, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 2, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:12:52.855 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83648, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2120, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489152, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036891648, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2170880, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 465198, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:12:52.856 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10397, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416931 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:12:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:12:52.858 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83648, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2120, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489152, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036891648, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2170880, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 465198, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:12:52.859 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10397, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416931 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:12:52.860 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 61700, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 100, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 300, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:12:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 100, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 100, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 100, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 100, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:12:52.862 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:12:52.863 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1220608, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 184566, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:12:52.864 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 16, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 16, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 2, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 2, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 154618822667 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672987 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 163208757258 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 1 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:12:52.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:12:52.867 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 5, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:12:03.186886+0000", 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 91, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 388, 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:12:52.868 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 529, 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:12:52.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-04-25T07:12:52.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-04-25T07:12:52.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-04-25T07:12:52.870 INFO:tasks.workunit.client.0.smithi031.stdout:writing non-divergent object existing_4 2024-04-25T07:12:52.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:12:53.196 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:12:53.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:12:53.197 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:12:53.197 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:12:53.197 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:12:53.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:12:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:12:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-04-25T07:12:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:12:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:12:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:12:53.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:12:53.316 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:12:53.316 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:12:53.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:12:53.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:12:53.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:12:53.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:12:53.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:12:53.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:12:53.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:12:53.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:12:53.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:12:53.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:12:53.849 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:12:53.849 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:12:53.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:12:53.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:53.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:12:54.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822670 2024-04-25T07:12:54.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822670 2024-04-25T07:12:54.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822670' 2024-04-25T07:12:54.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:54.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:12:54.168 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:12:54.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:12:54.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:12:54.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:12:54.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:54.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:12:54.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757261 2024-04-25T07:12:54.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757261 2024-04-25T07:12:54.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-04-25T07:12:54.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:54.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822670 2024-04-25T07:12:54.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:54.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:12:54.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822670 2024-04-25T07:12:54.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822670 2024-04-25T07:12:54.341 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 154618822670 2024-04-25T07:12:54.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-04-25T07:12:54.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:12:54.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822668 -lt 154618822670 2024-04-25T07:12:54.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:12:55.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:12:55.772 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:12:56.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822670 -lt 154618822670 2024-04-25T07:12:56.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:56.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757261 2024-04-25T07:12:56.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:56.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:12:56.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757261 2024-04-25T07:12:56.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:56.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757261 2024-04-25T07:12:56.274 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 163208757261 2024-04-25T07:12:56.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-04-25T07:12:56.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:12:56.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757261 -lt 163208757261 2024-04-25T07:12:56.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:12:56.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:56.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:57.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:12:57.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:57.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:57.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:57.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:57.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:57.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:57.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:57.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:57.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:57.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:57.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:12:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:12:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:12:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:58.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:12:58.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:12:58.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:58.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:58.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:58.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:58.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:58.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:58.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:58.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:12:58.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:58.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:58.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:12:59.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:12:59.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:12:59.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:12:59.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:12:59.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:12:59.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:12:59.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:12:59.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:13:00.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:00.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:00.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:00.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:00.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:00.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:00.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:00.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:00.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:00.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:00.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:00.630 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:01.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:01.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:01.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:01.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:01.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:01.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:13:01.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:13:01.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:13:02.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:02.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:02.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:02.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:02.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:02.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:02.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:02.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:03.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:03.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:03.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:13:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:13:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:13:04.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:04.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:04.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:04.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:05.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:05.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:05.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:05.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:05.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:05.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:05.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:05.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:05.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:05.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:05.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:05.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:05.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:13:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:13:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:13:07.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:07.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:07.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:07.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:07.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:07.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:07.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:07.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:08.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:08.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:08.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:08.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:08.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:08.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:08.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:08.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:08.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:08.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:08.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:08.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:08.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:13:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:13:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:13:12.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:12.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:12.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:12.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:12.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:12.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:12.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:12.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:12.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:12.980 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:12.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:12.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:13.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:13.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:13.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:13.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:14.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:14.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:14.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:13:14.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:13:14.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:13:20.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:20.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:20.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:20.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:20.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:20.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:21.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:21.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:22.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:22.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:22.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:13:22.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:13:22.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:13:29.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:13:29.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:29.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:29.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:29.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:29.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:29.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:29.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:29.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:13:29.822 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:29.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:29.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:30.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:13:30.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:13:30.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:13:30.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:13:30.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:13:30.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:13:30.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:13:31.506 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "f951f6d5-344d-487b-9fd6-0ae729298bf5", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:13:31.507211+0000", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:13:31.507 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.508 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 2 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:13:31.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.510 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:13:31.511 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:09:26.528021Z", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:09:26.528021Z", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:13:31.512 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.513 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:13:31.514 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.515 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 42, 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:09:27.464635+0000", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:12:14.138917+0000", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:12:11.248628+0000", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:12:13.909086+0000", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:13:31.516 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 9, 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:13:31.517 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:11:12.938535+0000", 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:13:31.518 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 1, 2024-04-25T07:13:31.519 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 1, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 1, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 1, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 1, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:13:31.520 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "29", 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:13:31.521 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:13:31.522 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T07:13:31.523 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 2, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 2, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.524 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "a136043e-f78a-4630-a55a-6bb7d421c30d", 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 5, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 36, 2024-04-25T07:13:31.525 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 41, 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 30, 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.526 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.527 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4293982427 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/4293982427", 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/4293982427", 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/4293982427", 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/4293982427", 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "9bf21071-871f-48fe-b320-e7acf88628d0", 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:13:31.530 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 23, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 31, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.531 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.533 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.534 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3314957454 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3314957454", 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3314957454", 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3314957454", 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3314957454", 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:13:31.535 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "6a0ee02d-b1be-478c-958f-d830803e2e77", 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 15, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 38, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 0, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 32, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:13:31.536 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:13:31.537 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:13:31.538 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:13:31.539 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2388680616 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.540 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/2388680616", 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/2388680616", 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2388680616", 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2388680616", 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.541 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:11:59.049882+0000", 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:10:00.252359+0000", 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.542 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:11:59.663915+0000", 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:10:35.072730+0000", 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:12:00.318626+0000", 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:13:31.543 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:11:08.853092+0000", 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:13:31.544 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:13:31.545 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.546 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/4293982427,v1:127.0.0.1:6805/4293982427]", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:13:31.547 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:09:34.207300Z", 2024-04-25T07:13:31.548 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/4293982427,v1:127.0.0.1:6803/4293982427]", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/4293982427,v1:127.0.0.1:6809/4293982427]", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/4293982427,v1:127.0.0.1:6807/4293982427]", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:13:31.549 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3314957454,v1:127.0.0.1:6813/3314957454]", 2024-04-25T07:13:31.550 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:13:31.551 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:10:07.981880Z", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:13:31.552 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3314957454,v1:127.0.0.1:6811/3314957454]", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3314957454,v1:127.0.0.1:6817/3314957454]", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3314957454,v1:127.0.0.1:6815/3314957454]", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:13:31.553 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/2388680616,v1:127.0.0.1:6813/2388680616]", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:13:31.554 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:13:31.555 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:10:42.218183Z", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/2388680616,v1:127.0.0.1:6811/2388680616]", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2388680616,v1:127.0.0.1:6817/2388680616]", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2388680616,v1:127.0.0.1:6815/2388680616]", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:13:31.556 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.557 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 36 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 38 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 42, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.560 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.561 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:13:31.562 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.563 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:13:31.564 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:13:31.564 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.564 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.564 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:13:31.564 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:13:31.564 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:13:31.565 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.566 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.567 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.568 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.569 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:13:31.570 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.571 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.572 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.573 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:13:31.574 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:13:31.575 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 0, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:13:31.576 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:13:31.577 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.578 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:13:31.579 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 1, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 62050, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 100, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 300, 2024-04-25T07:13:31.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 100, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 100, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 101, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 101, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:13:31.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:13:31.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:13:31.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 17, 2024-04-25T07:13:31.584 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 17, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 2, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 2, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83648, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2120, 2024-04-25T07:13:31.585 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489152, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036891648, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2170880, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 465898, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10397, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416931 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.586 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:13:31.587 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83648, 2024-04-25T07:13:31.588 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2120, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489152, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036891648, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2170880, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 465898, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10397, 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416931 2024-04-25T07:13:31.589 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:13:31.590 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 62050, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 100, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 300, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:13:31.591 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 100, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 100, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 101, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 101, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:13:31.592 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:13:31.593 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1220608, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 185266, 2024-04-25T07:13:31.594 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 17, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 17, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 2, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 2, 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:13:31.595 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 154618822677 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672987 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 163208757268 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:13:31.596 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 1 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T07:13:31.597 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:13:31.598 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 7, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:13:13.205503+0000", 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 111, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 451, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 612, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:13:31.599 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:13:31.600 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:13:31.600 INFO:tasks.workunit.client.0.smithi031.stderr:report 4201366498 2024-04-25T07:13:31.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:13:31.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-04-25T07:13:31.601 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery 2024-04-25T07:13:31.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:31.932 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:31.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 42'101 42:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:13:31.932 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:31.932 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:31.932 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:31.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:13:31.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-04-25T07:13:31.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:13:31.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:13:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:13:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:13:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:13:31.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:13:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:13:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:13:32.063 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:13:32.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:13:32.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-04-25T07:13:32.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:13:32.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:13:32.079 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:13:32.079 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:13:32.079 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:13:32.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:13:32.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:13:32.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:13:32.188 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:13:32.201 INFO:tasks.workunit.client.0.smithi031.stdout:revive divergent 1 2024-04-25T07:13:32.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-04-25T07:13:32.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:32.619 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:32.620 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 42'101 42:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:13:32.620 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:32.620 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:32.620 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:32.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-04-25T07:13:33.345 INFO:tasks.workunit.client.0.smithi031.stderr:noup is set 2024-04-25T07:13:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:13:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:13:33.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:13:33.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:13:33.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:13:33.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:13:33.368 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:13:33.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:13:33.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:13:33.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:13:33.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:13:33.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:13:33.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:13:33.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:13:33.411 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:33.410+0000 7f280367f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:33.441 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:33.442+0000 7f280367f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:33.443 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:33.442+0000 7f280367f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:33.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:13:33.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-04-25T07:13:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:33.994+0000 7f280367f7c0 -1 Falling back to public interface 2024-04-25T07:13:36.795 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:36.794+0000 7f280367f7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T07:13:38.828 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery divergent 2024-04-25T07:13:38.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-04-25T07:13:38.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:39.252 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:39.252 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:13:39.252 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:39.252 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:39.252 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:39.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:13:39.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:13:39.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:13:39.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:13:39.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:13:39.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:13:39.271 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:13:39.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:13:39.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:13:39.389 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:13:39.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-04-25T07:13:40.040 INFO:tasks.workunit.client.0.smithi031.stderr:noup is unset 2024-04-25T07:13:40.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-04-25T07:13:40.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:40.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:13:40.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:40.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:40.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:40.062 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:13:40.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:40.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:40.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:40.578 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/4293982427,v1:127.0.0.1:6803/4293982427] [v2:127.0.0.1:6804/4293982427,v1:127.0.0.1:6805/4293982427] exists,up a136043e-f78a-4630-a55a-6bb7d421c30d 2024-04-25T07:13:40.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:40.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:40.580 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:13:40.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:40.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:40.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:41.073 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1869418448,v1:127.0.0.1:6819/1869418448] [v2:127.0.0.1:6820/1869418448,v1:127.0.0.1:6821/1869418448] exists,up 9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:13:41.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:41.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:41.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:41.075 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:13:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:41.548 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2388680616,v1:127.0.0.1:6811/2388680616] [v2:127.0.0.1:6812/2388680616,v1:127.0.0.1:6813/2388680616] exists,up 6a0ee02d-b1be-478c-958f-d830803e2e77 2024-04-25T07:13:41.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:41.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:41.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:41.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:13:41.956 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:41.956 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:41.957 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:41.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-04-25T07:13:41.972 INFO:tasks.workunit.client.0.smithi031.stdout:wait for peering 2024-04-25T07:13:41.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:42.378 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:42.378 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:13:42.378 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:42.379 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:42.379 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:42.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-04-25T07:13:42.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-04-25T07:13:42.543 INFO:tasks.workunit.client.0.smithi031.stdout:killing divergent 1 2024-04-25T07:13:42.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:42.964 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:42.964 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:12:14.433131+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:13:48.364898+0000 0 0 2024-04-25T07:13:42.964 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:42.964 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:42.964 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:42.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:13:42.982 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:13:42.982 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:13:42.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:13:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:13:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:13:43.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:13:43.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.25959.out 2024-04-25T07:13:43.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.25959.out 2024-04-25T07:13:43.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:13:43.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:13:43.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:13:43.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:13:43.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:13:43.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.25959.out 2024-04-25T07:13:44.187 INFO:tasks.workunit.client.0.smithi031.stderr:Exporting 1.0 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=47/48 n=101 ec=23/23 lis/c=47/23 les/c/f=48/24/0 sis=47) 2024-04-25T07:13:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:023dd3d2:::existing_12:head# 2024-04-25T07:13:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:04c40a0c:::existing_93:head# 2024-04-25T07:13:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:064ff5fb:::existing_36:head# 2024-04-25T07:13:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:091f9f1d:::existing_94:head# 2024-04-25T07:13:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0a865a93:::existing_54:head# 2024-04-25T07:13:44.191 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0fed513c:::existing_41:head# 2024-04-25T07:13:44.191 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-04-25T07:13:44.192 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:169157fa:::existing_81:head# 2024-04-25T07:13:44.192 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:17731382:::existing_58:head# 2024-04-25T07:13:44.193 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18a45a39:::existing_43:head# 2024-04-25T07:13:44.193 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1c160f01:::existing_66:head# 2024-04-25T07:13:44.193 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1dd6c249:::existing_97:head# 2024-04-25T07:13:44.194 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:227e0c77:::existing_7:head# 2024-04-25T07:13:44.194 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:23e1b063:::existing_59:head# 2024-04-25T07:13:44.195 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:28566edb:::existing_78:head# 2024-04-25T07:13:44.195 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2a101182:::existing_20:head# 2024-04-25T07:13:44.196 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2d1519da:::existing_96:head# 2024-04-25T07:13:44.196 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3078b87a:::existing_86:head# 2024-04-25T07:13:44.196 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:30bde01b:::existing_15:head# 2024-04-25T07:13:44.197 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:317a7e19:::existing_56:head# 2024-04-25T07:13:44.197 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-04-25T07:13:44.198 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3e16d84b:::existing_48:head# 2024-04-25T07:13:44.198 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:45fe8203:::existing_75:head# 2024-04-25T07:13:44.198 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-04-25T07:13:44.201 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5298dcdb:::existing_57:head# 2024-04-25T07:13:44.201 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:52ebaa44:::existing_32:head# 2024-04-25T07:13:44.201 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:53588912:::existing_9:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5da0255c:::existing_13:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5da2dab8:::existing_53:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5ee3514b:::existing_30:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:602f83fe:::foo:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:60426cc7:::existing_62:head# 2024-04-25T07:13:44.202 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:628918ca:::existing_79:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:67e1780f:::existing_27:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:68143ca1:::existing_71:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6bac9509:::existing_29:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6bbec798:::existing_45:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6cb23a40:::existing_65:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7055ec5e:::existing_83:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:70a54e2b:::existing_87:head# 2024-04-25T07:13:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:750ace21:::existing_19:head# 2024-04-25T07:13:44.206 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:78623b1d:::existing_6:head# 2024-04-25T07:13:44.207 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79881152:::existing_63:head# 2024-04-25T07:13:44.207 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-04-25T07:13:44.207 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:81490677:::existing_76:head# 2024-04-25T07:13:44.207 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-04-25T07:13:44.208 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-04-25T07:13:44.208 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-04-25T07:13:44.208 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:923c5d4d:::existing_5:head# 2024-04-25T07:13:44.208 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:92731e71:::existing_89:head# 2024-04-25T07:13:44.209 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:928205cf:::existing_82:head# 2024-04-25T07:13:44.209 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:93be2267:::existing_24:head# 2024-04-25T07:13:44.210 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:99ceda2c:::existing_44:head# 2024-04-25T07:13:44.210 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9dd917ee:::existing_26:head# 2024-04-25T07:13:44.210 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a28805a6:::existing_72:head# 2024-04-25T07:13:44.211 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a294a8f6:::existing_16:head# 2024-04-25T07:13:44.211 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a2f5473a:::existing_42:head# 2024-04-25T07:13:44.212 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a504a533:::existing_90:head# 2024-04-25T07:13:44.212 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a691337e:::existing_61:head# 2024-04-25T07:13:44.212 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-04-25T07:13:44.213 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a9fde86e:::existing_18:head# 2024-04-25T07:13:44.213 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aa309faa:::existing_68:head# 2024-04-25T07:13:44.213 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-04-25T07:13:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b279b1fe:::existing_10:head# 2024-04-25T07:13:44.215 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b343e277:::existing_88:head# 2024-04-25T07:13:44.215 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b7130e90:::existing_80:head# 2024-04-25T07:13:44.215 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bc6f7309:::existing_70:head# 2024-04-25T07:13:44.215 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:be8489b7:::existing_38:head# 2024-04-25T07:13:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-04-25T07:13:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bebec877:::existing_74:head# 2024-04-25T07:13:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c04347d8:::existing_60:head# 2024-04-25T07:13:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c156033e:::existing_100:head# 2024-04-25T07:13:44.217 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-04-25T07:13:44.217 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:13:44.217 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:13:44.218 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-04-25T07:13:44.218 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ccb3865b:::existing_73:head# 2024-04-25T07:13:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cd84e19e:::existing_40:head# 2024-04-25T07:13:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-04-25T07:13:44.220 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d41a25c9:::existing_22:head# 2024-04-25T07:13:44.220 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d7f088da:::existing_91:head# 2024-04-25T07:13:44.220 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:13:44.220 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:13:44.221 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:13:44.221 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:13:44.222 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e0e22077:::existing_21:head# 2024-04-25T07:13:44.222 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e255b710:::existing_55:head# 2024-04-25T07:13:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:eff1d5db:::existing_14:head# 2024-04-25T07:13:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f1b156bf:::existing_23:head# 2024-04-25T07:13:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f5ce4199:::existing_34:head# 2024-04-25T07:13:44.224 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f6809456:::existing_11:head# 2024-04-25T07:13:44.224 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f699a2a9:::existing_67:head# 2024-04-25T07:13:44.224 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f9baac43:::existing_25:head# 2024-04-25T07:13:44.225 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb76688e:::existing_51:head# 2024-04-25T07:13:44.225 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fbb92eec:::existing_69:head# 2024-04-25T07:13:44.225 INFO:tasks.workunit.client.0.smithi031.stderr:Export successful 2024-04-25T07:13:44.226 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T07:13:44.226 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T07:13:44.226 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:13:44.428 INFO:tasks.workunit.client.0.smithi031.stderr:Remove successful 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.25959.out 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:13:44.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.25959.out 2024-04-25T07:13:45.742 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={23=1}} deleted_pools ) 2024-04-25T07:13:45.743 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T07:13:45.743 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:023dd3d2:::existing_12:head# 2024-04-25T07:13:45.743 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:45.806 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:04c40a0c:::existing_93:head# 2024-04-25T07:13:45.807 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:45.840 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:064ff5fb:::existing_36:head# 2024-04-25T07:13:45.840 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:45.873 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:091f9f1d:::existing_94:head# 2024-04-25T07:13:45.873 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:45.907 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a865a93:::existing_54:head# 2024-04-25T07:13:45.907 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:45.940 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0fed513c:::existing_41:head# 2024-04-25T07:13:45.940 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:45.974 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-04-25T07:13:45.974 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.007 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:169157fa:::existing_81:head# 2024-04-25T07:13:46.007 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.041 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17731382:::existing_58:head# 2024-04-25T07:13:46.041 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.074 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18a45a39:::existing_43:head# 2024-04-25T07:13:46.074 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.108 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1c160f01:::existing_66:head# 2024-04-25T07:13:46.108 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.141 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1dd6c249:::existing_97:head# 2024-04-25T07:13:46.141 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.175 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:227e0c77:::existing_7:head# 2024-04-25T07:13:46.175 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.225 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:23e1b063:::existing_59:head# 2024-04-25T07:13:46.225 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.258 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:28566edb:::existing_78:head# 2024-04-25T07:13:46.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.292 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2a101182:::existing_20:head# 2024-04-25T07:13:46.292 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.325 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2d1519da:::existing_96:head# 2024-04-25T07:13:46.325 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.359 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3078b87a:::existing_86:head# 2024-04-25T07:13:46.359 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.392 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:30bde01b:::existing_15:head# 2024-04-25T07:13:46.392 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.426 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:317a7e19:::existing_56:head# 2024-04-25T07:13:46.426 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.459 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-04-25T07:13:46.459 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.492 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3e16d84b:::existing_48:head# 2024-04-25T07:13:46.493 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.526 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:45fe8203:::existing_75:head# 2024-04-25T07:13:46.526 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.559 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-04-25T07:13:46.560 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.593 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5298dcdb:::existing_57:head# 2024-04-25T07:13:46.593 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.626 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:52ebaa44:::existing_32:head# 2024-04-25T07:13:46.626 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.660 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:53588912:::existing_9:head# 2024-04-25T07:13:46.660 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.693 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-04-25T07:13:46.693 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.726 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5da0255c:::existing_13:head# 2024-04-25T07:13:46.727 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.760 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5da2dab8:::existing_53:head# 2024-04-25T07:13:46.760 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.793 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5ee3514b:::existing_30:head# 2024-04-25T07:13:46.794 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.827 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:602f83fe:::foo:head# 2024-04-25T07:13:46.827 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.860 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:60426cc7:::existing_62:head# 2024-04-25T07:13:46.861 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.894 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:628918ca:::existing_79:head# 2024-04-25T07:13:46.894 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.927 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-04-25T07:13:46.927 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:46.961 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:67e1780f:::existing_27:head# 2024-04-25T07:13:46.961 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.003 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68143ca1:::existing_71:head# 2024-04-25T07:13:47.003 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.036 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6bac9509:::existing_29:head# 2024-04-25T07:13:47.036 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.069 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6bbec798:::existing_45:head# 2024-04-25T07:13:47.070 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.103 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6cb23a40:::existing_65:head# 2024-04-25T07:13:47.103 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.136 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7055ec5e:::existing_83:head# 2024-04-25T07:13:47.137 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.170 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:70a54e2b:::existing_87:head# 2024-04-25T07:13:47.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.203 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:750ace21:::existing_19:head# 2024-04-25T07:13:47.203 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.245 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:78623b1d:::existing_6:head# 2024-04-25T07:13:47.245 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.287 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79881152:::existing_63:head# 2024-04-25T07:13:47.287 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.329 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-04-25T07:13:47.329 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.371 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:81490677:::existing_76:head# 2024-04-25T07:13:47.371 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.404 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-04-25T07:13:47.404 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.438 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-04-25T07:13:47.438 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.473 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-04-25T07:13:47.473 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.532 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:923c5d4d:::existing_5:head# 2024-04-25T07:13:47.532 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.599 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:92731e71:::existing_89:head# 2024-04-25T07:13:47.599 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.633 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:928205cf:::existing_82:head# 2024-04-25T07:13:47.633 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.774 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:93be2267:::existing_24:head# 2024-04-25T07:13:47.775 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.808 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:99ceda2c:::existing_44:head# 2024-04-25T07:13:47.808 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.841 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9dd917ee:::existing_26:head# 2024-04-25T07:13:47.842 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.875 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a28805a6:::existing_72:head# 2024-04-25T07:13:47.875 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.908 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a294a8f6:::existing_16:head# 2024-04-25T07:13:47.908 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.942 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a2f5473a:::existing_42:head# 2024-04-25T07:13:47.942 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:47.975 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a504a533:::existing_90:head# 2024-04-25T07:13:47.975 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.009 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a691337e:::existing_61:head# 2024-04-25T07:13:48.009 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.042 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-04-25T07:13:48.042 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.076 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9fde86e:::existing_18:head# 2024-04-25T07:13:48.076 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.109 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aa309faa:::existing_68:head# 2024-04-25T07:13:48.109 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.142 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-04-25T07:13:48.143 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.218 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b279b1fe:::existing_10:head# 2024-04-25T07:13:48.218 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.251 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b343e277:::existing_88:head# 2024-04-25T07:13:48.251 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.285 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b7130e90:::existing_80:head# 2024-04-25T07:13:48.285 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.318 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bc6f7309:::existing_70:head# 2024-04-25T07:13:48.318 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.352 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:be8489b7:::existing_38:head# 2024-04-25T07:13:48.352 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.385 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-04-25T07:13:48.385 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.418 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bebec877:::existing_74:head# 2024-04-25T07:13:48.418 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.452 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c04347d8:::existing_60:head# 2024-04-25T07:13:48.452 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.485 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c156033e:::existing_100:head# 2024-04-25T07:13:48.485 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.519 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-04-25T07:13:48.519 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.552 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:13:48.552 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.586 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:13:48.586 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.619 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-04-25T07:13:48.619 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.653 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ccb3865b:::existing_73:head# 2024-04-25T07:13:48.653 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.686 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cd84e19e:::existing_40:head# 2024-04-25T07:13:48.686 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.719 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-04-25T07:13:48.720 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.753 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d41a25c9:::existing_22:head# 2024-04-25T07:13:48.753 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.786 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d7f088da:::existing_91:head# 2024-04-25T07:13:48.786 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.811 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:13:48.812 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.837 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:13:48.837 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.870 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:13:48.871 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.904 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:13:48.904 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.937 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e0e22077:::existing_21:head# 2024-04-25T07:13:48.937 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:48.971 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e255b710:::existing_55:head# 2024-04-25T07:13:48.971 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.004 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:eff1d5db:::existing_14:head# 2024-04-25T07:13:49.004 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.038 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f1b156bf:::existing_23:head# 2024-04-25T07:13:49.038 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.071 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f5ce4199:::existing_34:head# 2024-04-25T07:13:49.071 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.105 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6809456:::existing_11:head# 2024-04-25T07:13:49.105 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.138 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f699a2a9:::existing_67:head# 2024-04-25T07:13:49.138 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.171 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f9baac43:::existing_25:head# 2024-04-25T07:13:49.171 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.196 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb76688e:::existing_51:head# 2024-04-25T07:13:49.197 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.230 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fbb92eec:::existing_69:head# 2024-04-25T07:13:49.230 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:13:49.263 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 48 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=47/48 n=101 ec=23/23 lis/c=47/23 les/c/f=48/24/0 sis=47) 2024-04-25T07:13:49.331 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T07:13:49.866 INFO:tasks.workunit.client.0.smithi031.stdout:reviving divergent 1 2024-04-25T07:13:49.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-04-25T07:13:49.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:50.289 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:50.289 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T07:13:41.277219+0000 42'101 48:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:13:50.289 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:13:50.289 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:50.290 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:13:50.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:13:50.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:13:50.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:13:50.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:13:50.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:13:50.309 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:13:50.309 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:13:50.309 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:13:50.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:13:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:13:50.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:13:50.311 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:13:50.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:13:50.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:13:50.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:13:50.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:13:50.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:13:50.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:13:50.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:13:50.352 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:50.349+0000 7f47074ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:50.353+0000 7f47074ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:50.355 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:50.353+0000 7f47074ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:13:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:13:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:50.799 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:13:50.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:50.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:50.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:50.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:50.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:50.922 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:50.921+0000 7f47074ba7c0 -1 Falling back to public interface 2024-04-25T07:13:51.271 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1869418448,v1:127.0.0.1:6819/1869418448] [v2:127.0.0.1:6820/1869418448,v1:127.0.0.1:6821/1869418448] exists,up 9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:51.273 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:13:51.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:51.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:51.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:51.700 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1869418448,v1:127.0.0.1:6819/1869418448] [v2:127.0.0.1:6820/1869418448,v1:127.0.0.1:6821/1869418448] exists,up 9bf21071-871f-48fe-b320-e7acf88628d0 2024-04-25T07:13:51.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:51.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:51.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:51.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-04-25T07:13:53.769 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:13:53.765+0000 7f47074ba7c0 -1 osd.1 48 log_to_monitors true 2024-04-25T07:14:11.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:14:11.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:14:11.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:14:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:11.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:14:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:14:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:14:11.824 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:14:11.824 INFO:tasks.workunit.client.0.smithi031.stdout: "ops": [], 2024-04-25T07:14:11.824 INFO:tasks.workunit.client.0.smithi031.stdout: "num_ops": 0 2024-04-25T07:14:11.824 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:14:11.836 INFO:tasks.workunit.client.0.smithi031.stdout:allowing recovery 2024-04-25T07:14:11.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-04-25T07:14:11.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:12.254 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:12.255 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:13:55.302601+0000 48'102 52:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:28:57.603214+0000 0 0 2024-04-25T07:14:12.255 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:14:12.255 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:12.255 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:14:12.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:14:12.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:14:12.434 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:14:12.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:14:12.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:14:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:14:12.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:14:12.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:14:12.766 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:14:12.781 INFO:tasks.workunit.client.0.smithi031.stdout:reading divergent objects 2024-04-25T07:14:12.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-04-25T07:14:12.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:13.183 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:13.183 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:13:55.302601+0000 48'102 52:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:11:12.976703+0000 0'0 2024-04-25T07:11:12.976703+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:28:57.603214+0000 0 0 2024-04-25T07:14:13.183 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:14:13.183 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:13.183 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:14:13.199 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-04-25T07:14:13.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-04-25T07:14:13.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:14:13.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:14:13.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:14:13.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:14:13.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:14:13.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:14:13.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:14:13.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:14:13.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-04-25T07:14:13.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:14:13.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:14:13.414 INFO:tasks.workunit.client.0.smithi031.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:14:13.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:14:13.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:14:13.451 INFO:tasks.workunit.client.0.smithi031.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:14:13.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-04-25T07:14:13.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:14:13.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4627.0:1 2024-04-25T07:11:48.184584+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4627.0:1 2024-04-25T07:11:48.184584+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: more recent entry found: 42'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4750.0:1 2024-04-25T07:12:53.246597+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4619.0:1 2024-04-25T07:11:48.169128+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4619.0:1 2024-04-25T07:11:48.169128+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:11:48.192864+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:11:48.192864+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:14:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:11:48.192864+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2024-04-25T07:11:48.176238+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2024-04-25T07:11:48.176238+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-04-25T07:14:13.460 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2024-04-25T07:11:48.176238+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4632.0:1 2024-04-25T07:11:48.191898+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: keeping 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4632.0:1 2024-04-25T07:11:48.191898+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:14:13.461 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4632.0:1 2024-04-25T07:11:48.191898+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-04-25T07:11:48.179076+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-04-25T07:11:48.179076+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:14:13.462 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:13:41.198+0000 7f27e7609640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:14:13.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:14:13.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:14:13.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:14:13.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:14:13.470 INFO:tasks.workunit.client.0.smithi031.stdout:success 2024-04-25T07:14:13.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-04-25T07:14:13.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.25959.out 2024-04-25T07:14:13.471 INFO:tasks.workunit.client.0.smithi031.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.25959.out': No such file or directory 2024-04-25T07:14:13.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-04-25T07:14:13.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:14:13.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:14:14.035 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T07:14:14.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-04-25T07:14:14.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:14.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:14.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:14.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:14.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:14.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:14.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:14:14.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:14:14.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:14:14.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:14:14.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:14.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:14.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:14.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:14.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:14.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:14.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:14:14.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:14:14.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:14:14.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:14:14.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:14:14.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:14:14.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:14.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:14:14.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:14:14.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:14.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:14:14.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:14:14.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:14:14.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:14:14.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:14.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:14.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:14.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:14.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:14.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:14.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:14:14.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:14:14.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:14:14.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:14:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:14:14.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:14:14.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:14.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:14:14.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:14:14.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:14.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:14:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:14:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:14:14.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:14:14.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:14.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:14.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:14:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:14:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:14:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:14:14.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:14:14.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:14.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:14.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.25959 2024-04-25T07:14:14.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-04-25T07:14:14.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-04-25T07:14:14.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-04-25T07:14:14.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-04-25T07:14:14.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-04-25T07:14:14.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-04-25T07:14:14.422 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:14:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:14:14.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:14:14.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:14:14.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:14.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:14.886 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:14.886 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:14.886 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:14.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:14:14.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:14:15.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:14:15.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:14:15.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:14:15.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:14:15.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:14:15.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:14:15.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get fsid 2024-04-25T07:14:15.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:14:15.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:14:15.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:14:15.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:14:15.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:14:15.243 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:14:15.243 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:14:15.243 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:14:15.244 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:15.244 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:15.244 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:15.244 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:14:15.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:14:15.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get mon_host 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:14:15.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:14:15.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:14:15.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:15.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:15.925 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:15.925 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:15.925 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:15.925 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:14:15.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:14:15.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:14:15.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:15.988 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:14:15.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:14:15.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:14:15.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 2024-04-25T07:14:15.990 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 2024-04-25T07:14:15.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7' 2024-04-25T07:14:15.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:14:16.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIAipmVIj9ABAASd6qOfZZH8IvkVl+pNIjXQ== 2024-04-25T07:14:16.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIAipmVIj9ABAASd6qOfZZH8IvkVl+pNIjXQ=="}' 2024-04-25T07:14:16.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 -i td/divergent-priors/0/new.json 2024-04-25T07:14:16.316 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:14:16.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:14:16.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIAipmVIj9ABAASd6qOfZZH8IvkVl+pNIjXQ== --osd-uuid fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 2024-04-25T07:14:16.372 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.369+0000 7fd7c4e867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:16.384 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.385+0000 7fd7c4e867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:16.386 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.385+0000 7fd7c4e867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:16.386 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.385+0000 7fd7c4e867c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:14:16.386 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.385+0000 7fd7c4e867c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:14:16.387 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.385+0000 7fd7c4e867c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:14:16.424 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:16.425+0000 7fd7c4e867c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:14:22.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:14:22.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:14:22.520 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:14:22.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:14:22.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:14:23.153 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:14:23.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:14:23.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:14:23.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:14:23.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:14:23.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:14:23.196 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:23.197+0000 7f82cad447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:23.215 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:23.213+0000 7f82cad447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:23.216 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:23.217+0000 7f82cad447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:23.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:14:23.622 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:14:23.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:23.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:23.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:23.781+0000 7f82cad447c0 -1 Falling back to public interface 2024-04-25T07:14:24.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:25.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:25.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:25.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:14:25.087 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:14:25.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:25.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:25.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:26.220 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:26.217+0000 7f82cad447c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:14:26.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:26.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:26.571 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:14:26.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:14:26.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:26.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:27.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:28.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:28.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:28.043 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:14:28.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:14:28.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:28.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:28.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:29.486 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:14:29.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:14:29.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:29.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:29.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:30.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:30.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:30.987 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:14:30.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:14:30.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:30.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:31.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:32.431 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:14:32.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:14:32.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:32.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:32.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:33.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:33.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:33.973 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:14:33.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:14:33.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:33.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:34.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:35.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:35.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:35.417 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:14:35.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:14:35.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:35.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:35.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:36.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:36.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:36.970 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:14:36.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:14:36.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:36.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:37.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:38.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:38.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:38.620 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:14:38.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:14:38.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:38.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:39.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:40.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:40.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:40.133 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:14:40.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:14:40.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:40.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:41.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:42.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:42.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:42.113 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:14:42.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:14:42.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:42.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:42.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:43.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:43.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:43.862 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:14:43.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:14:43.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:43.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:44.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:45.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:45.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:45.514 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:14:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:14:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:46.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:47.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:47.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:47.652 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:14:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:14:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:48.217 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1798493433,v1:127.0.0.1:6803/1798493433] [v2:127.0.0.1:6804/1798493433,v1:127.0.0.1:6805/1798493433] exists,up fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 2024-04-25T07:14:48.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:14:48.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:14:48.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:14:48.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:14:48.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:14:48.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:14:48.224 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:14:48.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 696600d5-3d9f-4bee-9ecc-5fe9a71e1028' 2024-04-25T07:14:48.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:14:48.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoAipmB4/UDhAAsNatWD5OQSK9RxDRgsjtEQ== 2024-04-25T07:14:48.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoAipmB4/UDhAAsNatWD5OQSK9RxDRgsjtEQ=="}' 2024-04-25T07:14:48.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 -i td/divergent-priors/1/new.json 2024-04-25T07:14:48.754 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:14:48.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:14:48.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoAipmB4/UDhAAsNatWD5OQSK9RxDRgsjtEQ== --osd-uuid 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:14:48.819 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.817+0000 7fc9f7c307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:48.828 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.825+0000 7fc9f7c307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:48.830 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.829+0000 7fc9f7c307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:48.830 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.829+0000 7fc9f7c307c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:14:48.830 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.829+0000 7fc9f7c307c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:14:48.830 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.829+0000 7fc9f7c307c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:14:48.903 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:48.905+0000 7fc9f7c307c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:14:54.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:14:54.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:14:54.736 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:14:54.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:14:54.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:14:55.333 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:14:55.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:14:55.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:14:55.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:14:55.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:14:55.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:14:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:55.373+0000 7fa0bf9677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:55.383 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:55.381+0000 7fa0bf9677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:55.384 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:55.385+0000 7fa0bf9677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:55.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:14:55.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:14:55.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:55.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:14:55.982 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:55.981+0000 7fa0bf9677c0 -1 Falling back to public interface 2024-04-25T07:14:56.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:57.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:57.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:57.264 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:14:57.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:14:57.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:57.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:14:57.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:58.475 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:14:58.473+0000 7fa0bf9677c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:14:58.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:58.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:58.750 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:14:58.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:14:58.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:58.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:14:59.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:00.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:00.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:00.384 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:15:00.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:15:00.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:00.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:00.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:01.834 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:15:01.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:15:01.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:01.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:02.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:03.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:03.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:03.384 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:15:03.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:15:03.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:03.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:03.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:04.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:04.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:04.849 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:15:04.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:15:04.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:04.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:05.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:06.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:06.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:06.377 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:15:06.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:15:06.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:06.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:06.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:07.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:07.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:07.845 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:15:07.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:15:07.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:07.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:08.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:09.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:09.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:09.388 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:15:09.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:15:09.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:09.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:09.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:10.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:10.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:10.846 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:15:10.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:15:10.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:10.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:12.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:12.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:12.455 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:15:12.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:15:12.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:12.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:13.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:14.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:14.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:14.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:15:14.525 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:15:14.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:14.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:15.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:16.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:16.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:16.235 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:15:16.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:15:16.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:16.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:17.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:18.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:18.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:18.302 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:15:18.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:15:18.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:18.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:19.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:20.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:20.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:20.112 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:15:20.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:15:20.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:20.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:20.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:21.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:21.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:21.948 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:15:21.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:15:21.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:21.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:22.426 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3460899636,v1:127.0.0.1:6811/3460899636] [v2:127.0.0.1:6812/3460899636,v1:127.0.0.1:6813/3460899636] exists,up 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:15:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:15:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:15:22.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:15:22.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:15:22.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 2024-04-25T07:15:22.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0' 2024-04-25T07:15:22.433 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 2024-04-25T07:15:22.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:15:22.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKAypmhYKtGxAAh12rf8HL634RTvzz+Sb/yQ== 2024-04-25T07:15:22.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKAypmhYKtGxAAh12rf8HL634RTvzz+Sb/yQ=="}' 2024-04-25T07:15:22.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 -i td/divergent-priors/2/new.json 2024-04-25T07:15:23.027 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:15:23.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:15:23.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKAypmhYKtGxAAh12rf8HL634RTvzz+Sb/yQ== --osd-uuid 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 2024-04-25T07:15:23.094 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.092+0000 7fb4131907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:23.099 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.096+0000 7fb4131907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:23.101 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.100+0000 7fb4131907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:23.101 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.100+0000 7fb4131907c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:15:23.101 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.100+0000 7fb4131907c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:15:23.101 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.100+0000 7fb4131907c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:15:23.183 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:23.180+0000 7fb4131907c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:15:29.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:15:29.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:15:29.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:15:29.009 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:15:29.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:15:29.596 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:15:29.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:15:29.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:15:29.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:15:29.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:15:29.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:15:29.630 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:29.628+0000 7f83447287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:29.632 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:29.632+0000 7f83447287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:29.634 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:29.632+0000 7f83447287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:30.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:15:30.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:30.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:15:30.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:30.082 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:15:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:30.189 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:30.188+0000 7f83447287c0 -1 Falling back to public interface 2024-04-25T07:15:30.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:31.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:31.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:31.566 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:15:31.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:15:31.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:31.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:32.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:32.705 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:15:32.704+0000 7f83447287c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:15:33.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:33.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:33.059 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:15:33.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:15:33.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:33.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:33.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:34.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:34.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:34.503 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:15:34.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:15:34.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:34.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:35.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:36.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:36.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:36.007 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:15:36.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:15:36.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:36.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:36.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:37.565 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:15:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:15:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:38.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:39.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:39.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:39.027 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:15:39.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:15:39.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:39.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:39.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:40.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:40.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:40.470 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:15:40.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:15:40.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:40.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:41.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:41.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:41.997 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:15:41.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:15:41.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:41.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:42.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:43.522 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:15:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:15:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:43.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:44.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:44.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:44.990 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:15:44.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:15:44.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:44.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:45.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:46.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:46.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:46.862 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:15:46.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:15:46.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:46.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:47.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:48.481 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:15:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:15:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:49.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:50.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:50.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:50.207 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:15:50.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:15:50.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:50.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:50.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:51.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:51.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:51.996 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:15:51.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:15:51.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:51.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:53.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:53.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:53.710 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:15:53.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:15:53.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:53.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:54.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:55.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:55.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:55.724 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:15:55.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:15:55.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:55.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:56.185 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2439785199,v1:127.0.0.1:6819/2439785199] [v2:127.0.0.1:6820/2439785199,v1:127.0.0.1:6821/2439785199] exists,up 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 2024-04-25T07:15:56.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:56.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:56.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:56.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-04-25T07:15:56.893 INFO:tasks.workunit.client.0.smithi031.stderr:noout is set 2024-04-25T07:15:56.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-04-25T07:15:57.621 INFO:tasks.workunit.client.0.smithi031.stderr:noin is set 2024-04-25T07:15:57.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-04-25T07:15:58.433 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T07:15:58.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-04-25T07:15:58.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:15:59.112 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T07:15:59.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:16:00.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-04-25T07:16:00.754 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 3 2024-04-25T07:16:00.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-04-25T07:16:01.575 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 2 2024-04-25T07:16:01.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-04-25T07:16:02.380 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 pg_autoscale_mode to on 2024-04-25T07:16:02.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-04-25T07:16:02.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-04-25T07:16:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1714029362 2024-04-25T07:16:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-04-25T07:16:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-04-25T07:16:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-04-25T07:16:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:16:02.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:16:02.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:16:02.927 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:16:02.927 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:16:02.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:16:02.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:02.927 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:16:03.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T07:16:03.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T07:16:03.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T07:16:03.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:03.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:16:03.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672974 2024-04-25T07:16:03.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672974 2024-04-25T07:16:03.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-04-25T07:16:03.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:03.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:16:03.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509445 2024-04-25T07:16:03.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509445 2024-04-25T07:16:03.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509445' 2024-04-25T07:16:03.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:03.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T07:16:03.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:03.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:16:03.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T07:16:03.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:03.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T07:16:03.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T07:16:03.464 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836503 2024-04-25T07:16:03.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:16:03.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-04-25T07:16:03.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:16:04.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:16:04.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:16:05.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836503 2024-04-25T07:16:05.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:05.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672974 2024-04-25T07:16:05.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:05.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:16:05.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672974 2024-04-25T07:16:05.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:05.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672974 2024-04-25T07:16:05.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-04-25T07:16:05.446 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672974 2024-04-25T07:16:05.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:16:05.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672974 -lt 42949672974 2024-04-25T07:16:05.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:05.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509445 2024-04-25T07:16:05.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:05.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:16:05.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509445 2024-04-25T07:16:05.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:05.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509445 2024-04-25T07:16:05.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-04-25T07:16:05.940 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509445 2024-04-25T07:16:05.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:16:06.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509445 -lt 64424509445 2024-04-25T07:16:06.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-04-25T07:16:06.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:16:06.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:16:06.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:16:06.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:16:06.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:16:06.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:16:06.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:16:06.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:16:06.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:16:06.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:16:06.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:16:06.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:16:06.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:16:06.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:16:06.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:16:06.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:16:06.980 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:16:06.981 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:16:06.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:16:06.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:06.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:16:07.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-04-25T07:16:07.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-04-25T07:16:07.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-04-25T07:16:07.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:07.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:16:07.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T07:16:07.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T07:16:07.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672977' 2024-04-25T07:16:07.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:07.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:16:07.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-04-25T07:16:07.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-04-25T07:16:07.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672977 2-64424509448' 2024-04-25T07:16:07.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:07.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-04-25T07:16:07.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:07.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:16:07.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-04-25T07:16:07.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:07.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-04-25T07:16:07.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-04-25T07:16:07.475 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836506 2024-04-25T07:16:07.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:16:07.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836506 2024-04-25T07:16:07.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:16:08.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:16:08.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:16:09.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836506 2024-04-25T07:16:09.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:09.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T07:16:09.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:16:09.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T07:16:09.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:09.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T07:16:09.369 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672977 2024-04-25T07:16:09.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T07:16:09.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:16:09.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-25T07:16:09.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:09.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-04-25T07:16:09.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:09.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:16:09.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:09.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-04-25T07:16:09.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-04-25T07:16:09.837 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509448 2024-04-25T07:16:09.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-25T07:16:09.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:16:10.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509448 2024-04-25T07:16:10.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:16:10.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:10.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:10.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:16:10.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:10.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:10.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:10.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:10.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:10.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:10.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:16:11.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:11.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:16:11.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:16:11.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:16:11.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-04-25T07:16:11.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-04-25T07:16:12.276 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:16:12.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-04-25T07:16:12.293 INFO:tasks.workunit.client.0.smithi031.stdout:primary and soon to be divergent is 1 2024-04-25T07:16:12.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-04-25T07:16:12.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:16:12.687 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:12.687 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:15:59.955814+0000 0'0 31:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:18:43.474177+0000 0 0 2024-04-25T07:16:12.687 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:16:12.687 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:12.687 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:16:12.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-04-25T07:16:12.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1714029372 2024-04-25T07:16:12.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1714029372 - 1714029362 2024-04-25T07:16:12.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=10 2024-04-25T07:16:12.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 10 -gt 300 ']' 2024-04-25T07:16:12.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-04-25T07:16:12.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-04-25T07:16:12.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:16:12.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-04-25T07:16:12.708 INFO:tasks.workunit.client.0.smithi031.stdout:writing initial objects 2024-04-25T07:16:12.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-04-25T07:16:12.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-04-25T07:16:12.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:12.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-04-25T07:16:12.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:12.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-04-25T07:16:13.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-04-25T07:16:13.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-04-25T07:16:13.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-04-25T07:16:13.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-04-25T07:16:13.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-04-25T07:16:13.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-04-25T07:16:13.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:13.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-04-25T07:16:14.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:14.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-04-25T07:16:14.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:14.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-04-25T07:16:14.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:14.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-04-25T07:16:14.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:14.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-04-25T07:16:14.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:14.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-04-25T07:16:14.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:14.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-04-25T07:16:15.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:15.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-04-25T07:16:15.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:15.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-04-25T07:16:15.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:15.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-04-25T07:16:15.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:15.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-04-25T07:16:15.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:15.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-04-25T07:16:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-04-25T07:16:16.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:16.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-04-25T07:16:16.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:16.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-04-25T07:16:16.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:16.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-04-25T07:16:16.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:16.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-04-25T07:16:16.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:16.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-04-25T07:16:16.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:16.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-04-25T07:16:17.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:17.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-04-25T07:16:17.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:17.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-04-25T07:16:17.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:17.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-04-25T07:16:17.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:17.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-04-25T07:16:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-04-25T07:16:17.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:17.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-04-25T07:16:18.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:18.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-04-25T07:16:18.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:18.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-04-25T07:16:18.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:18.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-04-25T07:16:18.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:18.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-04-25T07:16:18.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:18.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-04-25T07:16:18.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:18.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-04-25T07:16:19.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:19.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-04-25T07:16:19.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:19.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-04-25T07:16:19.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:19.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-04-25T07:16:19.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:19.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-04-25T07:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-04-25T07:16:19.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:19.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-04-25T07:16:20.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-04-25T07:16:20.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-04-25T07:16:20.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-04-25T07:16:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-04-25T07:16:20.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-04-25T07:16:20.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-04-25T07:16:20.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:20.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-04-25T07:16:21.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:21.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-04-25T07:16:21.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:21.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-04-25T07:16:21.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:21.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-04-25T07:16:21.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:21.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-04-25T07:16:21.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:21.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-04-25T07:16:22.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:22.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-04-25T07:16:22.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:22.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-04-25T07:16:22.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:22.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-04-25T07:16:22.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:22.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-04-25T07:16:22.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:22.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-04-25T07:16:22.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:22.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-04-25T07:16:24.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:24.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-04-25T07:16:24.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:24.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-04-25T07:16:24.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:24.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-04-25T07:16:24.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:24.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-04-25T07:16:24.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:24.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-04-25T07:16:25.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:25.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-04-25T07:16:25.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:25.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-04-25T07:16:25.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:25.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-04-25T07:16:25.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:25.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-04-25T07:16:25.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:25.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-04-25T07:16:25.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:25.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-04-25T07:16:26.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:26.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-04-25T07:16:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-04-25T07:16:26.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:26.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-04-25T07:16:26.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:26.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-04-25T07:16:27.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:27.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-04-25T07:16:27.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:27.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-04-25T07:16:27.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:27.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-04-25T07:16:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-04-25T07:16:27.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:27.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-04-25T07:16:27.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:27.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-04-25T07:16:28.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:28.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-04-25T07:16:28.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:28.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-04-25T07:16:28.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:28.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-04-25T07:16:28.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:28.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-04-25T07:16:28.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:28.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-04-25T07:16:29.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:29.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-04-25T07:16:30.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:30.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-04-25T07:16:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-04-25T07:16:30.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:30.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-04-25T07:16:30.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:30.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-04-25T07:16:30.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:31.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-04-25T07:16:31.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:31.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-04-25T07:16:31.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:31.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-04-25T07:16:32.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:32.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-04-25T07:16:32.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:32.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-04-25T07:16:32.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:16:32.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-04-25T07:16:32.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:16:32.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-04-25T07:16:32.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:16:32.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:16:32.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:16:32.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:16:32.908 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:16:32.908 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:16:32.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:16:32.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:16:32.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:16:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:16:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:16:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:16:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:16:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:16:32.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:16:33.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836514 2024-04-25T07:16:33.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836514 2024-04-25T07:16:33.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836514' 2024-04-25T07:16:33.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:33.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672984 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672984 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836514 1-42949672984' 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:16:34.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T07:16:34.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T07:16:34.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836514 1-42949672984 2-64424509455' 2024-04-25T07:16:34.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:34.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836514 2024-04-25T07:16:34.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:34.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:16:34.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836514 2024-04-25T07:16:34.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836514 2024-04-25T07:16:34.225 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836514 2024-04-25T07:16:34.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-04-25T07:16:34.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:16:34.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836514 2024-04-25T07:16:34.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:16:35.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:16:35.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:16:36.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836514 2024-04-25T07:16:36.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:36.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672984 2024-04-25T07:16:36.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:36.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:16:36.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672984 2024-04-25T07:16:36.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:36.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672984 2024-04-25T07:16:36.355 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672984 2024-04-25T07:16:36.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-25T07:16:36.356 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:16:36.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-25T07:16:36.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:16:36.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T07:16:36.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:16:36.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:16:36.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T07:16:36.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:16:36.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T07:16:36.847 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509455 2024-04-25T07:16:36.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T07:16:36.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:16:37.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509455 2024-04-25T07:16:37.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:16:37.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:37.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:37.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:16:37.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:37.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:37.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:37.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:37.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:37.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:37.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:38.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-04-25T07:16:38.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:38.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:38.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:38.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-04-25T07:16:38.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' -1 2024-04-25T07:16:38.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:16:38.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=24 2024-04-25T07:16:38.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:38.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:16:39.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:39.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:39.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:39.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:39.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:39.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:39.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:39.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:39.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-04-25T07:16:39.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:39.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:39.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:40.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-04-25T07:16:40.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:16:40.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:16:40.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-04-25T07:16:40.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:16:40.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:16:40.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:16:40.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:16:40.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-04-25T07:16:40.525 INFO:tasks.workunit.client.0.smithi031.stdout:blackholing osds 0 2 2024-04-25T07:16:40.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-04-25T07:16:40.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:16:34.195816+0000 34'63 40:112 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:33.622344+0000 38'64 40:124 [0,1,2] 0 [0,1,2] 0 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:04:09.086650+0000 3 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:16:31.667423+0000 34'64 40:130 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:35:45.034386+0000 4 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:16:34.195548+0000 36'63 40:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-04-25T07:16:35.628805+0000 32'62 40:113 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:08:17.365512+0000 3 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 0 0 0 3085 0 0 16 48 16 active+clean 2024-04-25T07:16:32.531207+0000 34'64 40:131 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:20:54.730416+0000 5 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:35.676591+0000 36'64 40:126 [1,0,2] 1 [1,0,2] 1 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:55:02.178459+0000 3 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:36.588442+0000 32'62 40:119 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:44:08.666376+0000 2 0 2024-04-25T07:16:40.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-04-25T07:16:32.146807+0000 34'64 40:118 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-04-25T07:16:35.408022+0000 36'64 40:130 [2,0,1] 2 [2,0,1] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:53:20.529397+0000 6 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:27.537313+0000 32'62 40:124 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:29:14.433924+0000 2 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:16:34.195784+0000 38'63 40:115 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-04-25T07:16:32.513448+0000 34'63 40:124 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:23:50.279479+0000 1 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:34.663118+0000 34'63 40:125 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:17:12.000207+0000 3 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-04-25T07:16:30.623251+0000 34'64 40:130 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:00.144932+0000 2 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:32.665299+0000 32'62 40:124 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:06:50.078831+0000 2 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:24.480988+0000 36'63 40:125 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:11.392030+0000 2 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:34.449892+0000 34'64 40:131 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:48:49.139724+0000 3 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:16:38.451351+0000 34'64 40:126 [0,1,2] 0 [0,1,2] 0 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:14:51.805064+0000 4 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:16:34.194843+0000 34'63 40:112 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:33.621004+0000 36'63 40:125 [1,0,2] 1 [1,0,2] 1 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:24.895813+0000 3 0 2024-04-25T07:16:40.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:16:30.041039+0000 32'62 40:110 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-04-25T07:16:34.551985+0000 32'62 40:113 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:34.846194+0000 1 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-04-25T07:16:34.194957+0000 36'64 40:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:29.519667+0000 37'64 40:126 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:37:15.453379+0000 2 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:16:30.042391+0000 34'63 40:121 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:16:37.516519+0000 34'64 40:125 [0,1,2] 0 [0,1,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:21:08.245489+0000 4 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:16:34.196177+0000 32'62 40:111 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:31.673251+0000 38'64 40:126 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:33:42.022647+0000 2 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:33.492578+0000 32'62 40:118 [2,0,1] 2 [2,0,1] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:58:47.966247+0000 2 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-04-25T07:16:36.468666+0000 34'63 40:125 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:44:40.822505+0000 4 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-04-25T07:16:34.194724+0000 38'64 40:126 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:16:40.932 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:40.933 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:16:40.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:16:40.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:16:40.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:16:40.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:16:41.062 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:16:41.062 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:16:41.062 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:16:41.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:16:41.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:16:41.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:16:41.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:16:41.075 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:16:41.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:16:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:16:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:16:41.179 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:16:41.179 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:16:41.179 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:16:41.192 INFO:tasks.workunit.client.0.smithi031.stdout:create new divergent objects 2024-04-25T07:16:41.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-04-25T07:16:41.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-04-25T07:16:41.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:16:41.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-04-25T07:16:41.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:16:41.195 INFO:tasks.workunit.client.0.smithi031.stdout:writing divergent objects 2024-04-25T07:16:41.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-04-25T07:16:41.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-04-25T07:16:41.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:16:41.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-04-25T07:16:41.198 INFO:tasks.workunit.client.0.smithi031.stdout:remove divergent objects 2024-04-25T07:16:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:16:41.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-04-25T07:16:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:16:41.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-04-25T07:16:41.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-04-25T07:16:41.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:16:41.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-04-25T07:16:41.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-04-25T07:16:41.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-04-25T07:16:41.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-04-25T07:16:41.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-04-25T07:16:51.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-04-25T07:16:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:rados: no process found 2024-04-25T07:16:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-04-25T07:16:51.212 INFO:tasks.workunit.client.0.smithi031.stdout:killing all the osds 2024-04-25T07:16:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:39.626428+0000 34'63 45:134 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:02:43.237141+0000 3 0 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:46.491847+0000 38'64 45:130 [1,0,2] 1 [1,0,2] 1 38'64 2024-04-25T07:16:46.491644+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:29:08.799338+0000 3 0 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:16:31.667423+0000 34'64 50:157 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:07:51.286212+0000 4 0 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:40.536486+0000 36'63 45:131 [1,2,0] 1 [1,2,0] 1 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:39:52.941243+0000 3 0 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-04-25T07:16:35.628805+0000 32'62 47:132 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:11.836416+0000 3 0 2024-04-25T07:16:51.649 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 0 0 0 3435 0 0 17 48 17 active+clean 2024-04-25T07:16:32.531207+0000 42'65 45:146 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:28:35.357260+0000 5 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:35.676591+0000 36'64 45:140 [1,0,2] 1 [1,0,2] 1 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:08:16.182079+0000 3 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:36.588442+0000 32'62 45:133 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:47.192042+0000 2 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:16:45.598084+0000 34'64 45:129 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:45.597881+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:09.894444+0000 4 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-04-25T07:16:35.408022+0000 36'64 50:157 [2,0,1] 2 [2,0,1] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:13:43.416959+0000 6 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:27.537313+0000 32'62 45:138 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:01:03.934275+0000 2 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 active+clean 2024-04-25T07:16:43.550666+0000 38'63 45:137 [1,2,0] 1 [1,2,0] 1 38'63 2024-04-25T07:16:43.550454+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:05.740015+0000 5 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-04-25T07:16:32.513448+0000 34'63 47:143 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:13:37.479475+0000 1 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:34.663118+0000 34'63 45:139 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:54:03.555226+0000 3 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-04-25T07:16:30.623251+0000 34'64 50:157 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:42:50.210625+0000 2 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:32.665299+0000 32'62 45:138 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:52:09.447897+0000 2 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:24.480988+0000 36'63 45:139 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:13:18.751331+0000 2 0 2024-04-25T07:16:51.650 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:34.449892+0000 34'64 50:158 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:00:45.289618+0000 3 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 0 0 0 1851 0 0 18 48 18 active+clean 2024-04-25T07:16:43.451434+0000 42'66 45:145 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:47:24.253932+0000 4 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-04-25T07:16:38.567846+0000 34'63 45:134 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:40:36.301909+0000 4 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:33.621004+0000 36'63 45:139 [1,0,2] 1 [1,0,2] 1 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:46:11.838283+0000 3 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 0 0 0 1234 0 0 15 48 15 active+clean 2024-04-25T07:16:36.379298+0000 42'63 50:146 [2,0,1] 2 [2,0,1] 2 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:23:47.946242+0000 3 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-04-25T07:16:34.551985+0000 32'62 47:132 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:39:13.376603+0000 1 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:42.545887+0000 36'64 45:142 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:42.545682+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:41.522009+0000 3 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:29.519667+0000 37'64 45:140 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:53:31.544728+0000 2 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:16:37.423041+0000 34'63 50:156 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:02:07.034668+0000 3 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:16:44.542550+0000 34'64 45:129 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:44.542214+0000 34'64 2024-04-25T07:16:44.542214+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:43:13.880565+0000 4 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-04-25T07:16:41.566513+0000 32'62 45:133 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:41.566300+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:54:52.906968+0000 1 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:16:31.673251+0000 38'64 45:140 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:11:34.670451+0000 2 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-04-25T07:16:33.492578+0000 32'62 45:132 [2,0,1] 2 [2,0,1] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:15.783032+0000 2 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-04-25T07:16:36.468666+0000 34'63 47:144 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:45:19.093857+0000 4 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 6 0 0 0 0 3435 0 0 18 48 18 active+clean 2024-04-25T07:16:37.722192+0000 42'66 45:150 [1,2,0] 1 [1,2,0] 1 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:57:11.278084+0000 5 0 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:16:51.651 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:51.652 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:16:51.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:16:51.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:16:51.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:16:51.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:16:51.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:16:51.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:16:51.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:16:51.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:16:51.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-04-25T07:16:52.343 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:16:52.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:16:52.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-04-25T07:16:53.045 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:16:53.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:16:53.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-04-25T07:16:53.639 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:16:53.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:16:53.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-04-25T07:16:54.192 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T07:16:54.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:16:54.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-04-25T07:16:54.754 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already out. 2024-04-25T07:16:54.770 INFO:tasks.workunit.client.0.smithi031.stdout:bringing up non_divergent 0 2 2024-04-25T07:16:54.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-04-25T07:16:54.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:16:55.194 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:55.194 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:39.626428+0000 34'63 45:134 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:02:43.237141+0000 3 0 2024-04-25T07:16:55.194 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:46.491847+0000 38'64 45:130 [1,0,2] 1 [1,0,2] 1 38'64 2024-04-25T07:16:46.491644+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:29:08.799338+0000 3 0 2024-04-25T07:16:55.194 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:31.667423+0000 34'64 50:157 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:07:51.286212+0000 4 0 2024-04-25T07:16:55.194 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:40.536486+0000 36'63 45:131 [1,2,0] 1 [1,2,0] 1 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:39:52.941243+0000 3 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:35.628805+0000 32'62 47:132 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:11.836416+0000 3 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 0 0 0 3435 0 0 17 48 17 stale+active+clean 2024-04-25T07:16:32.531207+0000 42'65 45:146 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:28:35.357260+0000 5 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:35.676591+0000 36'64 45:140 [1,0,2] 1 [1,0,2] 1 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:08:16.182079+0000 3 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:36.588442+0000 32'62 45:133 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:47.192042+0000 2 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 0 0 0 2468 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:45.598084+0000 34'64 45:129 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:45.597881+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:09.894444+0000 4 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:35.408022+0000 36'64 50:157 [2,0,1] 2 [2,0,1] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:13:43.416959+0000 6 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:27.537313+0000 32'62 45:138 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:01:03.934275+0000 2 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:43.550666+0000 38'63 45:137 [1,2,0] 1 [1,2,0] 1 38'63 2024-04-25T07:16:43.550454+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:05.740015+0000 5 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:32.513448+0000 34'63 47:143 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:13:37.479475+0000 1 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:34.663118+0000 34'63 45:139 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:54:03.555226+0000 3 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:30.623251+0000 34'64 50:157 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:42:50.210625+0000 2 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:32.665299+0000 32'62 45:138 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:52:09.447897+0000 2 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:24.480988+0000 36'63 45:139 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:13:18.751331+0000 2 0 2024-04-25T07:16:55.195 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:34.449892+0000 34'64 50:158 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:00:45.289618+0000 3 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 0 0 0 1851 0 0 18 48 18 stale+active+clean 2024-04-25T07:16:43.451434+0000 42'66 45:145 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:47:24.253932+0000 4 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:38.567846+0000 34'63 45:134 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:40:36.301909+0000 4 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:33.621004+0000 36'63 45:139 [1,0,2] 1 [1,0,2] 1 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:46:11.838283+0000 3 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 0 0 0 1234 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:36.379298+0000 42'63 50:146 [2,0,1] 2 [2,0,1] 2 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:23:47.946242+0000 3 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:34.551985+0000 32'62 47:132 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:39:13.376603+0000 1 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:42.545887+0000 36'64 45:142 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:42.545682+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:41.522009+0000 3 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:29.519667+0000 37'64 45:140 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:53:31.544728+0000 2 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:37.423041+0000 34'63 50:156 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:02:07.034668+0000 3 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:44.542550+0000 34'64 45:129 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:44.542214+0000 34'64 2024-04-25T07:16:44.542214+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:43:13.880565+0000 4 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 1 0 0 0 0 617 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:41.566513+0000 32'62 45:133 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:41.566300+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:54:52.906968+0000 1 0 2024-04-25T07:16:55.196 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-04-25T07:16:31.673251+0000 38'64 45:140 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:11:34.670451+0000 2 0 2024-04-25T07:16:55.197 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-04-25T07:16:33.492578+0000 32'62 45:132 [2,0,1] 2 [2,0,1] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:15.783032+0000 2 0 2024-04-25T07:16:55.197 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 stale+active+clean 2024-04-25T07:16:36.468666+0000 34'63 47:144 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:45:19.093857+0000 4 0 2024-04-25T07:16:55.197 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 6 0 0 0 0 3435 0 0 18 48 18 stale+active+clean 2024-04-25T07:16:37.722192+0000 42'66 45:150 [1,2,0] 1 [1,2,0] 1 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:57:11.278084+0000 5 0 2024-04-25T07:16:55.197 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:16:55.197 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:55.197 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:16:55.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:16:55.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-04-25T07:16:55.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:16:55.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:16:55.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:16:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:16:55.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:16:55.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:16:55.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:16:55.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:16:55.218 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:16:55.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:16:55.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:16:55.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:16:55.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:16:55.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:16:55.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:16:55.256 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:16:55.259+0000 7fb77a4d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:55.258 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:16:55.259+0000 7fb77a4d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:55.259 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:16:55.259+0000 7fb77a4d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:55.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:16:55.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:55.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:16:55.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:55.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:55.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:55.630 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:16:55.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:55.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:55.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:55.829 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:16:55.831+0000 7fb77a4d07c0 -1 Falling back to public interface 2024-04-25T07:16:56.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:57.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:57.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:57.109 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:16:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:16:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:57.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:58.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:58.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:58.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:16:58.637 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:16:58.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:58.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:58.894 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:16:58.895+0000 7fb77a4d07c0 -1 osd.0 41 log_to_monitors true 2024-04-25T07:16:59.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:00.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:00.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:00.092 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:17:00.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:17:00.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:00.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:00.570 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up out weight 0 up_from 62 up_thru 37 down_at 52 last_clean_interval [5,41) [v2:127.0.0.1:6802/3700950424,v1:127.0.0.1:6803/3700950424] [v2:127.0.0.1:6804/3700950424,v1:127.0.0.1:6805/3700950424] exists,up fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 2024-04-25T07:17:00.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:17:00.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:17:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:17:00.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:17:00.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:17:00.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:17:00.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:17:00.577 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:17:00.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:17:00.618 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:17:00.619+0000 7fbbe78b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:00.621 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:17:00.623+0000 7fbbe78b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:00.622 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:17:00.623+0000 7fbbe78b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:17:01.051 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:17:01.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:01.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:17:01.181 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:17:01.183+0000 7fbbe78b37c0 -1 Falling back to public interface 2024-04-25T07:17:01.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:02.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:02.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:02.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:17:02.557 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:17:02.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:02.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:17:03.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:04.044 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:17:04.047+0000 7fbbe78b37c0 -1 osd.2 41 log_to_monitors true 2024-04-25T07:17:04.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:04.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:04.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:17:04.047 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:17:04.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:04.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:17:04.549 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up out weight 0 up_from 66 up_thru 35 down_at 55 last_clean_interval [15,41) [v2:127.0.0.1:6810/3768691028,v1:127.0.0.1:6811/3768691028] [v2:127.0.0.1:6812/3768691028,v1:127.0.0.1:6813/3768691028] exists,up 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 2024-04-25T07:17:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:17:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:17:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:17:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:17:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-04-25T07:17:05.340 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. 2024-04-25T07:17:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:17:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-04-25T07:17:06.371 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already in. 2024-04-25T07:17:06.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:17:06.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-04-25T07:17:06.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:17:06.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:17:06.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:17:06.395 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:17:06.396 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:17:06.396 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:17:06.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:17:06.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:17:06.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:17:06.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:17:06.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:17:06.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:17:06.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:17:06.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:17:06.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:17:07.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:17:07.049 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:17:07.049 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:17:07.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:17:07.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:17:07.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:17:07.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=266287972356 2024-04-25T07:17:07.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 266287972356 2024-04-25T07:17:07.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-266287972356' 2024-04-25T07:17:07.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:17:07.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:17:07.514 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:17:07.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:17:07.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:17:07.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:17:07.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:17:07.516 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:17:07.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=283467841539 2024-04-25T07:17:07.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 283467841539 2024-04-25T07:17:07.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-266287972356 2-283467841539' 2024-04-25T07:17:07.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:17:07.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-266287972356 2024-04-25T07:17:07.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:17:07.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:17:07.754 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-266287972356 2024-04-25T07:17:07.754 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:17:07.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=266287972356 2024-04-25T07:17:07.756 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 266287972356 2024-04-25T07:17:07.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 266287972356' 2024-04-25T07:17:07.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:17:08.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 266287972354 -lt 266287972356 2024-04-25T07:17:08.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:17:09.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:17:09.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:17:09.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 266287972356 -lt 266287972356 2024-04-25T07:17:09.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:17:09.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-283467841539 2024-04-25T07:17:09.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:17:09.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:17:09.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-283467841539 2024-04-25T07:17:09.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:17:09.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=283467841539 2024-04-25T07:17:09.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2024-04-25T07:17:09.703 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 283467841539 2024-04-25T07:17:09.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:17:10.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 283467841539 -lt 283467841539 2024-04-25T07:17:10.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:17:10.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:10.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:10.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:17:10.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:10.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:10.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:10.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:10.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:10.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:10.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:11.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:11.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:11.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:11.156 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:11.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:11.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:17:11.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:17:11.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:17:11.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:11.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:17:11.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:11.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:11.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:11.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:11.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:11.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:11.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:11.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:12.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:12.246 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:12.246 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:12.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:12.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:13.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:13.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:13.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:17:13.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:13.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:17:13.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:13.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:13.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:13.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:13.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:13.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:13.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:13.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:13.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:13.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:13.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:13.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:14.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:14.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:14.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:14.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:14.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:14.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:14.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:14.604 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:14.605 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:15.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:15.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:15.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:17:15.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:15.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:17:15.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:15.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:15.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:15.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:15.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:15.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:15.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:15.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:16.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:16.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:16.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:16.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:16.611 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:16.612 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:17.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:17.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:17.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:17:17.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:17.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:17:17.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:17.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:17.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:17.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:17.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:17.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:17.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:17.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:18.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:18.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:18.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:18.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:19.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:19.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:19.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:19.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:19.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:19.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:17:19.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:19.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:17:21.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:21.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:21.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:21.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:21.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:21.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:21.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:21.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:21.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:21.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:21.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:21.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:22.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:22.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:22.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:22.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:22.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:22.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:22.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:22.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:22.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:22.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:22.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:22.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:17:22.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:22.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:17:25.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:25.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:25.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:25.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:25.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:25.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:25.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:25.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:26.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:26.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:26.399 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:26.399 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:26.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:26.971 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:27.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:27.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:27.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:17:27.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:27.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:17:33.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:33.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:33.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:33.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:33.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:33.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:33.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:33.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:34.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:34.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:34.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:34.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:34.982 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:34.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:35.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:35.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:35.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:17:35.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:35.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:17:42.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:42.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:42.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:42.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:42.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:42.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:42.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:42.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:43.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:43.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:43.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:43.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:43.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:43.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:43.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:43.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:43.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:44.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:44.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:44.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:17:44.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:17:45.015 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:17:45.015 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "74146037-9e2f-4cbd-a6c7-3fb43d802c3d", 2024-04-25T07:17:45.015 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:17:45.016601+0000", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:17:45.016 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.017 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 96/288 objects degraded (33.333%), 32 pgs degraded", 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 32 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.018 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.019 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.020 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.021 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.022 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.023 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.024 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.026 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:14:14.455726Z", 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:14:14.455726Z", 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:17:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:17:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.029 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.030 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 75, 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:14:15.408189+0000", 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:17:07.351168+0000", 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:17:04.385186+0000", 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:17:05.784286+0000", 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:17:45.031 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 9, 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:17:45.032 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:15:58.795533+0000", 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:17:45.033 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 32, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 19, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 32, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 32, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 32, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:17:45.034 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "74", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:17:45.035 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:17:45.036 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T07:17:45.037 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 1.7799999713897705, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 1.7799999713897705, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 1.190000057220459, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 1.190000057220459, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.038 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7", 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 5, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 41, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 62, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 71, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 52, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.039 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:17:45.040 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.041 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.042 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/3700950424", 2024-04-25T07:17:45.043 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/3700950424", 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3700950424", 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3700950424", 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "696600d5-3d9f-4bee-9ecc-5fe9a71e1028", 2024-04-25T07:17:45.044 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 50, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 53, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.045 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.046 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.047 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.048 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3460899636", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3460899636", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3460899636", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3460899636", 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:17:45.049 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "5f420cff-f1b7-4d23-ae57-c5125f4ae9c0", 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 15, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 41, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 66, 2024-04-25T07:17:45.050 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 73, 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 55, 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.051 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.052 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.053 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:17:45.054 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3768691028", 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3768691028", 2024-04-25T07:17:45.055 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3768691028", 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3768691028", 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:16:52.154564+0000", 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:17:45.056 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:14:47.445432+0000", 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:16:52.750326+0000", 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:15:20.595355+0000", 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.057 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:16:53.473103+0000", 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:15:55.181697+0000", 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:17:45.058 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:17:45.059 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/3700950424,v1:127.0.0.1:6805/3700950424]", 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:17:45.060 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:17:45.061 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:14:21.812015Z", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/3700950424,v1:127.0.0.1:6803/3700950424]", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3700950424,v1:127.0.0.1:6809/3700950424]", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3700950424,v1:127.0.0.1:6807/3700950424]", 2024-04-25T07:17:45.062 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:17:45.063 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3460899636,v1:127.0.0.1:6813/3460899636]", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:17:45.064 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:14:54.050682Z", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:17:45.065 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3460899636,v1:127.0.0.1:6811/3460899636]", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3460899636,v1:127.0.0.1:6817/3460899636]", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3460899636,v1:127.0.0.1:6815/3460899636]", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:17:45.066 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3768691028,v1:127.0.0.1:6813/3768691028]", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:17:45.067 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:17:45.068 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:15:28.332693Z", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3768691028,v1:127.0.0.1:6811/3768691028]", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3768691028,v1:127.0.0.1:6817/3768691028]", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3768691028,v1:127.0.0.1:6815/3768691028]", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:17:45.069 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:17:45.070 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 62 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 67 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.071 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 75, 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.072 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:17:45.073 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.074 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:17:45.075 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.076 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stderr:report 3884801458 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:17:45.077 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:17:45.078 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:17:45.079 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:17:45.080 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:17:45.081 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:17:45.082 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.083 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.084 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:17:45.085 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 0, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.086 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:17:45.087 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:17:45.088 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 32, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 32, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 59155, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 96, 2024-04-25T07:17:45.089 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 288, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 96, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 96, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 55, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 55, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:17:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:17:45.091 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:17:45.092 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 486, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 486, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 64, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 64, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:17:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 96, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 84452, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2776, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81589, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314488348, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036068352, 2024-04-25T07:17:45.094 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2842624, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 711001, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83548004 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:17:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 123, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 123, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 123000000, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 123000000 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:17:45.096 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 96, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 84452, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2776, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81589, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314488348, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036068352, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2842624, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 711001, 2024-04-25T07:17:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83548004 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 123, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 123, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 123000000, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 123000000 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 32, 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:17:45.099 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 59155, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 96, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 288, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 96, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 96, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 55, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 55, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:17:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:17:45.101 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1220608, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 184566, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:17:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 486, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 486, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 64, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 64, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 266287972363 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.103 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672987 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 283467841546 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 32 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 19, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 32, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 13, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 32, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:17:45.105 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 5, 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:16:54.755129+0000", 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:17:45.106 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 94, 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 433, 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 590, 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:17:45.107 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:17:45.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:17:45.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-04-25T07:17:45.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-04-25T07:17:45.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-04-25T07:17:45.108 INFO:tasks.workunit.client.0.smithi031.stdout:writing non-divergent object existing_4 2024-04-25T07:17:45.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 75:127 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 75:125 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 75:133 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 75:117 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767949+0000 0 0 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 75:105 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.431 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 75:141 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 75:118 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 75:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 75:145 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 75:145 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767971+0000 0 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 75:117 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 75:135 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 75:119 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767899+0000 0 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 75:134 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767937+0000 0 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 75:132 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 75:141 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:17:45.432 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 34'64 75:113 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 75:126 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 75:124 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767960+0000 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767925+0000 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 75:130 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 75:133 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 75:136 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 75:139 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 75:142 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 75:116 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 75:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 75:131 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 75:117 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:17:45.433 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 75:134 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767914+0000 0 0 2024-04-25T07:17:45.434 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:17:45.434 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:17:45.434 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:17:45.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:17:45.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:17:45.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-04-25T07:17:45.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:17:45.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:17:45.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:17:45.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:17:45.591 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:17:45.592 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:17:45.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:17:45.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:17:45.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:17:45.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:17:45.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:17:45.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:17:45.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:17:45.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:17:45.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:17:46.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:17:46.150 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:17:46.150 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:17:46.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:17:46.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:17:46.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:17:46.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=266287972366 2024-04-25T07:17:46.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 266287972366 2024-04-25T07:17:46.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-266287972366' 2024-04-25T07:17:46.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:17:46.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:17:46.445 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:17:46.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:17:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:17:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:17:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:17:46.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:17:46.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=283467841549 2024-04-25T07:17:46.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 283467841549 2024-04-25T07:17:46.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-266287972366 2-283467841549' 2024-04-25T07:17:46.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:17:46.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-266287972366 2024-04-25T07:17:46.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:17:46.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:17:46.611 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-266287972366 2024-04-25T07:17:46.611 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:17:46.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=266287972366 2024-04-25T07:17:46.612 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 266287972366 2024-04-25T07:17:46.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 266287972366' 2024-04-25T07:17:46.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:17:47.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 266287972366 -lt 266287972366 2024-04-25T07:17:47.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:17:47.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:17:47.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-283467841549 2024-04-25T07:17:47.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:17:47.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-283467841549 2024-04-25T07:17:47.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:17:47.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=283467841549 2024-04-25T07:17:47.091 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 283467841549 2024-04-25T07:17:47.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 283467841549' 2024-04-25T07:17:47.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:17:47.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 283467841549 -lt 283467841549 2024-04-25T07:17:47.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:17:47.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:47.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:48.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:17:48.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:48.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:48.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:48.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:48.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:48.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:48.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:48.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:48.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:48.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:48.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:49.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:49.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:17:49.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:17:49.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:17:49.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:49.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:17:49.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:49.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:49.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:49.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:49.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:49.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:49.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:49.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:50.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:50.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:50.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:50.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:50.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:50.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:50.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:50.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:17:50.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:50.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:17:51.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:51.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:51.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:51.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:51.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:51.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:51.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:51.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:51.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:51.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:51.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:52.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:52.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:52.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:17:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:52.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:17:53.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:53.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:53.045 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:53.045 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:53.045 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:53.045 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:53.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:53.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:53.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:53.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:53.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:53.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:54.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:54.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:54.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:54.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:54.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:17:54.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:54.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:17:55.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:55.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:55.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:55.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:55.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:55.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:55.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:55.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:55.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:55.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:55.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:55.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:56.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:56.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:56.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:56.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:56.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:56.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:56.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:56.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:56.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:17:57.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:17:57.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:17:58.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:17:58.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:17:58.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:17:58.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:17:58.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:17:58.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:17:58.611 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:17:58.611 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:17:59.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:17:59.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:17:59.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:17:59.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:17:59.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:17:59.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:17:59.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:18:00.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:18:00.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:18:00.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:18:00.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:18:00.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:18:03.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:18:03.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:18:03.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:18:03.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:18:03.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:18:03.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:18:03.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:18:03.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:18:03.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:18:03.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:18:03.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:03.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:18:04.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:18:04.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:18:04.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:18:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:18:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:18:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:18:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:18:04.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:18:11.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:18:11.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:18:11.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:18:11.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:18:11.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:18:11.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:18:11.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:18:11.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:18:11.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:18:11.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:18:11.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:11.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:18:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:18:12.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:18:12.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:18:13.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:18:13.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:18:13.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:18:13.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:18:13.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:18:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:18:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:18:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:18:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:18:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:18:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:18:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:18:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:18:20.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:18:20.830 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:18:20.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:20.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:21.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:18:21.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:18:21.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:18:21.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:18:21.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:18:21.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:18:21.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:18:21.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:18:21.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:18:21.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:18:21.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:18:21.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:18:21.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "74146037-9e2f-4cbd-a6c7-3fb43d802c3d", 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:18:22.501552+0000", 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:18:22.499 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:18:22.500 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 96/288 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-04-25T07:18:22.501 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 64 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.2 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.4 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.5 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.6 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.7 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.8 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.503 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.9 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.b is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.c is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.d is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.504 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.e is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.10 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.11 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.505 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.12 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.13 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.14 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.15 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.16 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.506 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.17 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.18 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.19 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1b is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.507 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1c is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1e is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:18:22.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.509 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:14:14.455726Z", 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:14:14.455726Z", 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:18:22.510 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:18:22.511 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.512 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 75, 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:18:22.513 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:14:15.408189+0000", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:17:07.351168+0000", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:17:04.385186+0000", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:17:05.784286+0000", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:18:22.514 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 9, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:18:22.515 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:15:58.795533+0000", 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:18:22.516 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 32, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 19, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 32, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 32, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 32, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.517 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "74", 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:18:22.518 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:18:22.519 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 1.7799999713897705, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 1.7799999713897705, 2024-04-25T07:18:22.520 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 1.190000057220459, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 1.190000057220459, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7", 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:18:22.521 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 5, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 41, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 62, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 71, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 52, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:18:22.522 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.523 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.524 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3700950424 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/3700950424", 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/3700950424", 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3700950424", 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3700950424", 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "696600d5-3d9f-4bee-9ecc-5fe9a71e1028", 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:18:22.526 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 50, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 53, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.528 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.529 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3460899636 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.530 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3460899636", 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3460899636", 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3460899636", 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3460899636", 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "5f420cff-f1b7-4d23-ae57-c5125f4ae9c0", 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:18:22.531 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 15, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 41, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 66, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 73, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 55, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:18:22.532 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.534 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3768691028 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.535 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3768691028", 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3768691028", 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3768691028", 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3768691028", 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.536 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:16:52.154564+0000", 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:14:47.445432+0000", 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:16:52.750326+0000", 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:18:22.537 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:15:20.595355+0000", 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:16:53.473103+0000", 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:15:55.181697+0000", 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.538 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.539 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.540 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/3700950424,v1:127.0.0.1:6805/3700950424]", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:18:22.541 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:14:21.812015Z", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:18:22.542 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/3700950424,v1:127.0.0.1:6803/3700950424]", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3700950424,v1:127.0.0.1:6809/3700950424]", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3700950424,v1:127.0.0.1:6807/3700950424]", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:18:22.543 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3460899636,v1:127.0.0.1:6813/3460899636]", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:18:22.544 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:18:22.545 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:14:54.050682Z", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3460899636,v1:127.0.0.1:6811/3460899636]", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3460899636,v1:127.0.0.1:6817/3460899636]", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3460899636,v1:127.0.0.1:6815/3460899636]", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:18:22.546 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:18:22.547 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3768691028,v1:127.0.0.1:6813/3768691028]", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:18:22.548 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:15:28.332693Z", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:18:22.549 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3768691028,v1:127.0.0.1:6811/3768691028]", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3768691028,v1:127.0.0.1:6817/3768691028]", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3768691028,v1:127.0.0.1:6815/3768691028]", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:18:22.550 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 62 2024-04-25T07:18:22.551 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 67 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 75, 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:18:22.552 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:18:22.553 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.554 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:18:22.555 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.556 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.557 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:18:22.560 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:18:22.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.562 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:18:22.563 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:18:22.564 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 0, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:18:22.565 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:18:22.566 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.567 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 32, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 32, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 59505, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 96, 2024-04-25T07:18:22.568 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 288, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 96, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 96, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 56, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 56, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:18:22.569 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:18:22.570 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:18:22.571 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 487, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 487, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 64, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 64, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:18:22.572 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 96, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 84416, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2776, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81589, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314488384, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:18:22.573 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036105216, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2842624, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 711701, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83548004 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:18:22.574 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 123, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 123, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 123000000, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 123000000 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:18:22.575 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 96, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 84416, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2776, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81589, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314488384, 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:18:22.576 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036105216, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2842624, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 711701, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83548004 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:18:22.577 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 123, 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 123, 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 123000000, 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 123000000 2024-04-25T07:18:22.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 32, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 59505, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 96, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 288, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:18:22.579 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 96, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 96, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 56, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 56, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:18:22.580 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:18:22.581 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1220608, 2024-04-25T07:18:22.582 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 185266, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 487, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 487, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 64, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 64, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:18:22.583 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 266287972373 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672987 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 283467841556 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 32 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.584 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 19, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 32, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 13, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 32, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:18:22.585 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 7, 2024-04-25T07:18:22.586 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:18:06.781823+0000", 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 112, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 496, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 671, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:18:22.587 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:18:22.588 INFO:tasks.workunit.client.0.smithi031.stderr:report 400792513 2024-04-25T07:18:22.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:18:22.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-04-25T07:18:22.588 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery 2024-04-25T07:18:22.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 75:127 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 75:125 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 75:133 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 75:117 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767949+0000 0 0 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 75:105 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.928 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 75:141 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 75:118 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 75:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 75:145 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 75:145 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767971+0000 0 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 75:117 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 75:135 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 75:119 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767899+0000 0 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 75:134 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767937+0000 0 0 2024-04-25T07:18:22.929 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 75:132 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 75:141 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 75'65 75:114 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:50.090118+0000 0 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 75:126 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 75:124 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767960+0000 0 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767925+0000 0 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 75:130 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 75:133 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 75:136 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 75:139 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 75:142 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:18:22.930 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 75:116 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 75:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 75:131 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 75:117 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 75:134 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767914+0000 0 0 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:22.931 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:22.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:18:22.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:18:22.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:18:22.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:18:22.945 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:18:22.945 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:22.945 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:18:22.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:18:22.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:18:22.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:18:23.058 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:18:23.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:18:23.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:18:23.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:18:23.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:18:23.073 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:18:23.073 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:23.073 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:18:23.074 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:18:23.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:18:23.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:18:23.189 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:18:23.203 INFO:tasks.workunit.client.0.smithi031.stdout:revive divergent 1 2024-04-25T07:18:23.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-04-25T07:18:23.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:23.610 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:23.610 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 75:127 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.610 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 75:125 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.610 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 75:133 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 75:117 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767949+0000 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 75:105 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 75:141 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 75:118 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 75:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 75:145 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 75:145 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767971+0000 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 75:117 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 75:135 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 75:119 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767899+0000 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 75:134 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767937+0000 0 0 2024-04-25T07:18:23.611 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 75:132 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 75:141 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 75'65 75:114 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:50.090118+0000 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 75:126 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 75:124 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767960+0000 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 75:107 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767925+0000 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 75:130 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 75:133 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 75:136 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 75:139 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 75:142 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 75:116 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 75:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 75:131 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 75:117 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:23.612 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 75:134 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:17:12.767914+0000 0 0 2024-04-25T07:18:23.613 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:23.613 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:23.613 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:23.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-04-25T07:18:24.374 INFO:tasks.workunit.client.0.smithi031.stderr:noup is set 2024-04-25T07:18:24.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:18:24.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:18:24.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:18:24.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:18:24.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:18:24.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:18:24.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:18:24.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:18:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:18:24.404 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:18:24.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:18:24.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:24.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:18:24.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:18:24.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:18:24.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:18:24.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:18:24.448 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:24.450+0000 7ff2b71be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:24.450 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:24.450+0000 7ff2b71be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:24.452 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:24.454+0000 7ff2b71be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:24.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:18:24.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-04-25T07:18:25.037 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:25.038+0000 7ff2b71be7c0 -1 Falling back to public interface 2024-04-25T07:18:28.039 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:28.038+0000 7ff2b71be7c0 -1 osd.1 51 log_to_monitors true 2024-04-25T07:18:29.967 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery divergent 2024-04-25T07:18:29.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-04-25T07:18:29.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:30.399 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:30.399 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 77:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359514+0000 0 0 2024-04-25T07:18:30.399 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 77:129 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359475+0000 0 0 2024-04-25T07:18:30.399 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 77:137 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 76:119 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 77:109 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359416+0000 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 77:122 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359554+0000 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 77:125 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359436+0000 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 76:147 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 77:149 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 76:109 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 77:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359494+0000 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 76:137 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 76:121 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 77:138 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:18:30.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 76:109 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 76:134 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 75'65 77:118 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359533+0000 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 77:130 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359593+0000 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 76:126 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 76:109 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 76:132 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 77:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359455+0000 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 76:138 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 77:143 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 76:144 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 77:120 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359574+0000 0 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 77:135 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 77:135 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:18:30.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 77:121 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359391+0000 0 0 2024-04-25T07:18:30.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 76:136 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:30.402 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:30.402 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:30.402 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:30.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:18:30.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:18:30.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:18:30.417 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:18:30.417 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:30.418 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:18:30.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:18:30.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:18:30.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:18:30.534 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:18:30.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-04-25T07:18:31.212 INFO:tasks.workunit.client.0.smithi031.stderr:noup is unset 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:31.236 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:18:31.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:31.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:31.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:18:32.000 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 62 up_thru 71 down_at 52 last_clean_interval [5,41) [v2:127.0.0.1:6802/3700950424,v1:127.0.0.1:6803/3700950424] [v2:127.0.0.1:6804/3700950424,v1:127.0.0.1:6805/3700950424] exists,up fe96d976-f0a8-4398-bf54-b4ae4fe7f4b7 2024-04-25T07:18:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:18:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:18:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:32.002 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:18:32.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:32.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:32.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:32.484 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 53 last_clean_interval [10,51) [v2:127.0.0.1:6818/1223931720,v1:127.0.0.1:6819/1223931720] [v2:127.0.0.1:6820/1223931720,v1:127.0.0.1:6821/1223931720] exists,up 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:18:32.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:32.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:32.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:32.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-04-25T07:18:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:18:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:32.486 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:18:32.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:32.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:32.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:32.956 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 66 up_thru 80 down_at 55 last_clean_interval [15,41) [v2:127.0.0.1:6810/3768691028,v1:127.0.0.1:6811/3768691028] [v2:127.0.0.1:6812/3768691028,v1:127.0.0.1:6813/3768691028] exists,up 5f420cff-f1b7-4d23-ae57-c5125f4ae9c0 2024-04-25T07:18:32.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:32.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:32.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:32.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 77:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359514+0000 0 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 77:129 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359475+0000 0 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 77:137 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 78:123 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970155+0000 0 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 77:109 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359416+0000 0 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 77:122 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359554+0000 0 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 77:125 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359436+0000 0 0 2024-04-25T07:18:33.383 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 78:151 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 77:149 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970207+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 77:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359494+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 78:141 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 78:125 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970026+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 77:138 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970133+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 78:138 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 75'65 77:118 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359533+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 77:130 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359593+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 78:130 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970177+0000 0 0 2024-04-25T07:18:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970109+0000 0 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 78:136 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 77:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359455+0000 0 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 78:142 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 77:143 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 78:148 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 77:120 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359574+0000 0 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 77:135 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 77:135 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 77:121 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359391+0000 0 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 78:140 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970073+0000 0 0 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:33.385 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:33.400 INFO:tasks.workunit.client.0.smithi031.stdout:wait for peering 2024-04-25T07:18:33.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-04-25T07:18:33.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:33.819 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:33.819 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 77:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359514+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 77:129 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359475+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 77:137 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 78:123 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970155+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 77:109 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359416+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 77:122 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359554+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 77:125 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359436+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 78:151 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 77:149 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970207+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 77:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359494+0000 0 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 78:141 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:18:33.820 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 78:125 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970026+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 77:138 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970133+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 78:138 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 75'65 77:118 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359533+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 77:130 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359593+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 78:130 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970177+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970109+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 78:136 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 77:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359455+0000 0 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 78:142 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 77:143 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:18:33.821 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 78:148 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 77:120 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359574+0000 0 0 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 77:135 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 77:135 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 77:121 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359391+0000 0 0 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 78:140 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970073+0000 0 0 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:33.822 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:33.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-04-25T07:18:33.977 INFO:tasks.workunit.client.0.smithi031.stdout:killing divergent 1 2024-04-25T07:18:33.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-04-25T07:18:33.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 2 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545711+0000 34'63 77:131 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359514+0000 0 0 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.849081+0000 38'64 77:129 [2,0] 2 [2,0] 2 38'64 2024-04-25T07:16:33.622144+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359475+0000 0 0 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545610+0000 34'64 77:137 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:44.032480+0000 4 0 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869385+0000 36'63 78:123 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970155+0000 0 0 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.847361+0000 32'62 77:109 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359416+0000 0 0 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 5 0 5 0 0 3085 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.869290+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:34:36.337698+0000 5 0 2024-04-25T07:18:34.400 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545891+0000 36'64 77:122 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359554+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.846468+0000 32'62 77:125 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359436+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870352+0000 34'64 78:151 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:39.459487+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:34.098835+0000 4 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.848333+0000 36'64 77:149 [2,0] 2 [2,0] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:23.874733+0000 6 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872275+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970207+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.511711+0000 38'63 77:121 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:25.541419+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359494+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.874475+0000 34'63 78:141 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:40.440765+0000 1 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.871213+0000 34'63 78:125 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970026+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.846754+0000 34'64 77:138 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:49.907746+0000 2 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872444+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970133+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.870533+0000 36'63 78:138 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:13.842611+0000 2 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.545145+0000 34'64 77:145 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:57:08.914390+0000 3 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-04-25T07:17:07.545152+0000 75'65 77:118 [2,0] 2 [2,0] 2 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359533+0000 0 0 2024-04-25T07:18:34.401 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.863975+0000 34'63 77:130 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359593+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.869976+0000 36'63 78:130 [0,2] 0 [0,2] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970177+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.873385+0000 32'62 78:113 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970109+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 1 0 0 617 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.872057+0000 32'62 78:136 [0,2] 0 [0,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:40:45.584223+0000 1 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.870177+0000 36'64 77:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359455+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872271+0000 37'64 78:142 [0,2] 0 [0,2] 0 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:26.314068+0000 2 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:07.545085+0000 34'63 77:143 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:34.186728+0000 3 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.875231+0000 34'64 78:148 [0,2] 0 [0,2] 0 34'64 2024-04-25T07:16:37.516272+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:09:39.571072+0000 4 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:07.546533+0000 32'62 77:120 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:15:58.847017+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359574+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:07.546034+0000 38'64 77:135 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:56:45.661650+0000 2 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 2 0 0 1234 0 0 14 48 14 active+undersized+degraded 2024-04-25T07:17:06.845851+0000 32'62 77:135 [2,0] 2 [2,0] 2 32'62 2024-04-25T07:16:33.492365+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:32.769739+0000 2 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-04-25T07:17:06.845944+0000 34'63 77:121 [2,0] 2 [2,0] 2 34'63 2024-04-25T07:16:36.468519+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:26.359391+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-04-25T07:17:06.872334+0000 38'64 78:140 [0,2] 0 [0,2] 0 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:29.970073+0000 0 0 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:34.402 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:34.403 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:34.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:18:34.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:18:34.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:18:34.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:18:34.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:18:34.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:18:34.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.25959.out 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.25959.out 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:18:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.25959.out 2024-04-25T07:18:35.754 INFO:tasks.workunit.client.0.smithi031.stderr:Exporting 1.1b info 1.1b( v 38'64 lc 0'0 (31'48,38'64] local-lis/les=80/81 n=4 ec=33/23 lis/c=80/39 les/c/f=81/40/0 sis=80) 2024-04-25T07:18:35.755 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:18:35.760 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:18:35.761 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:18:35.767 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:18:35.768 INFO:tasks.workunit.client.0.smithi031.stderr:Export successful 2024-04-25T07:18:35.769 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T07:18:35.769 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T07:18:35.769 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-04-25T07:18:35.870 INFO:tasks.workunit.client.0.smithi031.stderr:Remove successful 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.25959.out 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:18:36.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.25959.out 2024-04-25T07:18:37.311 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e82 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-04-25T07:18:37.311 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.1b 2024-04-25T07:18:37.311 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:18:37.311 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:18:37.382 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:18:37.382 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:18:37.424 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:18:37.424 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:18:37.465 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:18:37.466 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T07:18:37.507 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 81 info 1.1b( v 38'64 lc 0'0 (31'48,38'64] local-lis/les=80/81 n=4 ec=33/23 lis/c=80/39 les/c/f=81/40/0 sis=80) 2024-04-25T07:18:37.549 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T07:18:38.086 INFO:tasks.workunit.client.0.smithi031.stdout:reviving divergent 1 2024-04-25T07:18:38.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-04-25T07:18:38.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:38.511 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:38.511 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:32.059931+0000 34'63 82:147 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:37.268661+0000 0 0 2024-04-25T07:18:38.511 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:32.063268+0000 38'64 82:144 [1,2,0] 1 [1,2,0] 1 38'64 2024-04-25T07:16:46.491644+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:46:34.890773+0000 3 0 2024-04-25T07:18:38.511 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:32.056488+0000 34'64 82:153 [2,1,0] 2 [2,1,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:03:08.294945+0000 4 0 2024-04-25T07:18:38.511 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:32.164911+0000 36'63 82:134 [1,0,2] 1 [1,0,2] 1 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.511 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-04-25T07:18:32.063946+0000 32'62 82:138 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:40:01.066409+0000 3 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 4 1 1 0 0 2430 0 0 16 48 16 active+recovery_wait+degraded 2024-04-25T07:18:32.167182+0000 34'64 82:160 [1,2,0] 1 [1,2,0] 1 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-04-25T07:18:32.057157+0000 36'64 82:165 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:37.144219+0000 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-04-25T07:18:32.166819+0000 32'62 82:133 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:32.060167+0000 34'64 82:143 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:45.597881+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:29:39.191410+0000 4 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-04-25T07:18:32.066291+0000 36'64 82:165 [2,0,1] 2 [2,0,1] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:59:30.284188+0000 6 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-04-25T07:18:32.166582+0000 32'62 82:123 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:32.056736+0000 38'63 82:148 [1,2,0] 1 [1,2,0] 1 38'63 2024-04-25T07:16:43.550454+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:37.144204+0000 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-04-25T07:18:32.541460+0000 34'63 82:153 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:45:10.058207+0000 1 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:32.166538+0000 34'63 82:139 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-04-25T07:18:32.060615+0000 34'64 82:154 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:53:53.455361+0000 2 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:32.058091+0000 32'62 82:133 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:37.144244+0000 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:18:32.062453+0000 36'63 82:130 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:46:58.566463+0000 2 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:32.056731+0000 34'64 82:161 [2,1,0] 2 [2,1,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:39:23.832557+0000 3 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 1 1 0 0 2780 0 0 17 48 17 active+recovering+degraded 2024-04-25T07:18:32.064705+0000 75'65 82:156 [1,2,0] 1 [1,2,0] 1 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:18:38.512 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:32.065299+0000 34'63 82:146 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:37.268730+0000 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:32.475366+0000 36'63 82:142 [0,2,1] 0 [0,2,1] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 1 0 0 1195 0 0 14 48 14 active+recovery_wait+degraded 2024-04-25T07:18:32.558987+0000 32'62 82:124 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-04-25T07:18:32.474503+0000 32'62 82:148 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:38:05.110003+0000 1 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-04-25T07:18:32.166964+0000 36'64 82:148 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:42.545682+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:32.058085+0000 37'64 82:134 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:37.752403+0000 2 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:18:32.059259+0000 34'63 82:159 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:37:15.254771+0000 3 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:32.055997+0000 34'64 82:140 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:44.542214+0000 34'64 2024-04-25T07:16:44.542214+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:01:13.658723+0000 4 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:32.057387+0000 32'62 82:139 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:41.566300+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:37.144232+0000 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:32.056662+0000 38'64 82:165 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:46:52.859241+0000 2 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:32.164442+0000 32'62 82:133 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:49.496906+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-04-25T07:18:32.060649+0000 34'63 82:138 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:51.503641+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:23:53.523447+0000 4 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 1 1 0 0 2429 0 0 16 48 16 active+recovery_wait+degraded 2024-04-25T07:18:32.166247+0000 38'64 82:154 [1,0,2] 1 [1,0,2] 1 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:18:38.513 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:38.514 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:18:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:18:38.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:18:38.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:18:38.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:38.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:18:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:18:38.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:18:38.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:18:38.533 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:18:38.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:18:38.568 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:38.570+0000 7f2936e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:38.570 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:38.570+0000 7f2936e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:38.571 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:38.574+0000 7f2936e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:39.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:18:39.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:39.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:18:39.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:39.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:39.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:39.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:39.050 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:18:39.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:39.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:39.145 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:39.146+0000 7f2936e707c0 -1 Falling back to public interface 2024-04-25T07:18:39.514 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 53 last_clean_interval [10,51) [v2:127.0.0.1:6818/1223931720,v1:127.0.0.1:6819/1223931720] [v2:127.0.0.1:6820/1223931720,v1:127.0.0.1:6821/1223931720] exists,up 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:18:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:39.516 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:18:39.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:39.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:39.999 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 53 last_clean_interval [10,51) [v2:127.0.0.1:6818/1223931720,v1:127.0.0.1:6819/1223931720] [v2:127.0.0.1:6820/1223931720,v1:127.0.0.1:6821/1223931720] exists,up 696600d5-3d9f-4bee-9ecc-5fe9a71e1028 2024-04-25T07:18:40.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:40.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:40.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:40.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-04-25T07:18:41.887 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:18:41.886+0000 7f2936e707c0 -1 osd.1 82 log_to_monitors true 2024-04-25T07:19:00.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:19:00.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:19:00.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:19:00.004 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:00.004 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:00.004 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:00.004 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:19:00.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:19:00.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:19:00.125 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:19:00.125 INFO:tasks.workunit.client.0.smithi031.stdout: "ops": [], 2024-04-25T07:19:00.125 INFO:tasks.workunit.client.0.smithi031.stdout: "num_ops": 0 2024-04-25T07:19:00.125 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:19:00.138 INFO:tasks.workunit.client.0.smithi031.stdout:allowing recovery 2024-04-25T07:19:00.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-04-25T07:19:00.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:43.687902+0000 34'63 87:161 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.716826+0000 38'64 87:154 [1,2,0] 1 [1,2,0] 1 38'64 2024-04-25T07:16:46.491644+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:08:33.686369+0000 3 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:43.684318+0000 34'64 87:167 [2,1,0] 2 [2,1,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:38:29.692390+0000 4 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:43.716999+0000 36'63 87:144 [1,0,2] 1 [1,0,2] 1 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-04-25T07:18:43.718328+0000 32'62 87:148 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:19:09.655844+0000 3 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 4 1 1 0 0 2430 0 0 16 48 16 active+recovery_wait+degraded 2024-04-25T07:18:43.722814+0000 34'64 87:169 [1,2,0] 1 [1,2,0] 1 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-04-25T07:18:43.712502+0000 36'64 87:175 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-04-25T07:18:43.717856+0000 32'62 87:143 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.556 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 5 0 0 0 0 3085 0 0 17 48 17 active+clean 2024-04-25T07:18:43.710108+0000 82'65 87:155 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:45.597881+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:41:43.455017+0000 4 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-04-25T07:18:43.713404+0000 36'64 87:179 [2,0,1] 2 [2,0,1] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:39.103565+0000 6 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-04-25T07:18:43.721846+0000 32'62 87:133 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:43.681806+0000 38'63 87:158 [1,2,0] 1 [1,2,0] 1 38'63 2024-04-25T07:16:43.550454+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-04-25T07:18:43.687131+0000 34'63 87:166 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:07.940573+0000 1 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:43.716910+0000 34'63 87:149 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-04-25T07:18:43.717307+0000 34'64 87:168 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:54:56.904166+0000 2 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:43.716033+0000 32'62 87:143 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:18:43.715899+0000 36'63 87:140 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:06:26.260806+0000 2 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.683845+0000 34'64 87:175 [2,1,0] 2 [2,1,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:35:05.986933+0000 3 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 1 1 0 0 2780 0 0 17 48 17 active+recovery_wait+degraded 2024-04-25T07:18:43.682851+0000 75'65 87:164 [1,2,0] 1 [1,2,0] 1 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:43.687953+0000 34'63 87:160 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.557 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:43.689311+0000 36'63 87:155 [0,2,1] 0 [0,2,1] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:48.781063+0000 0 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 1 0 0 1195 0 0 14 48 14 active+recovering+degraded 2024-04-25T07:18:43.720802+0000 32'62 87:137 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-04-25T07:18:43.715551+0000 32'62 87:161 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:08:37.757307+0000 1 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-04-25T07:18:43.722391+0000 36'64 87:158 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:42.545682+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.682010+0000 37'64 87:144 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:44:49.281033+0000 2 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:18:43.654685+0000 34'63 87:173 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:25:49.398896+0000 3 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:43.682956+0000 34'64 87:150 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:44.542214+0000 34'64 2024-04-25T07:16:44.542214+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:44:45.271403+0000 4 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:43.714974+0000 32'62 87:149 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:41.566300+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.720313+0000 38'64 87:175 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:35:04.052387+0000 2 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:43.720248+0000 32'62 87:143 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:49.496906+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-04-25T07:18:43.716312+0000 34'63 87:148 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:51.503641+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:22:28.266260+0000 4 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 1 1 0 0 2429 0 0 16 48 16 active+recovery_wait+degraded 2024-04-25T07:18:43.715516+0000 38'64 87:163 [1,0,2] 1 [1,0,2] 1 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:19:00.558 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:00.559 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:19:00.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:19:00.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:19:00.728 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:19:00.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:19:00.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:19:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:19:00.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:19:00.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:19:01.144 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:19:01.159 INFO:tasks.workunit.client.0.smithi031.stdout:reading divergent objects 2024-04-25T07:19:01.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-04-25T07:19:01.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.1f 2 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:43.687902+0000 34'63 87:161 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:39.626323+0000 34'63 2024-04-25T07:16:39.626323+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.1e 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.716826+0000 38'64 87:154 [1,2,0] 1 [1,2,0] 1 38'64 2024-04-25T07:16:46.491644+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:08:33.686369+0000 3 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:43.684318+0000 34'64 87:167 [2,1,0] 2 [2,1,0] 2 34'64 2024-04-25T07:16:31.667201+0000 34'64 2024-04-25T07:16:31.667201+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:38:29.692390+0000 4 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:43.716999+0000 36'63 87:144 [1,0,2] 1 [1,0,2] 1 36'63 2024-04-25T07:16:40.536261+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-04-25T07:18:43.718328+0000 32'62 87:148 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:35.628588+0000 32'62 2024-04-25T07:16:35.628588+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:19:09.655844+0000 3 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.9 4 1 1 0 0 2430 0 0 16 48 16 active+recovery_wait+degraded 2024-04-25T07:18:43.722814+0000 34'64 87:169 [1,2,0] 1 [1,2,0] 1 34'64 2024-04-25T07:16:32.530973+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.8 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-04-25T07:18:43.712502+0000 36'64 87:175 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:35.676215+0000 36'64 2024-04-25T07:16:35.676215+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-04-25T07:18:43.717856+0000 32'62 87:143 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:36.588295+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 5 0 0 0 0 3085 0 0 17 48 17 active+clean 2024-04-25T07:18:43.710108+0000 82'65 87:155 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:45.597881+0000 34'64 2024-04-25T07:16:39.459487+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:41:43.455017+0000 4 0 2024-04-25T07:19:01.568 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-04-25T07:18:43.713404+0000 36'64 87:179 [2,0,1] 2 [2,0,1] 2 36'64 2024-04-25T07:16:35.407815+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:39.103565+0000 6 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-04-25T07:18:43.721846+0000 32'62 87:133 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:27.537114+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:43.681806+0000 38'63 87:158 [1,2,0] 1 [1,2,0] 1 38'63 2024-04-25T07:16:43.550454+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-04-25T07:18:43.687131+0000 34'63 87:166 [0,1,2] 0 [0,1,2] 0 34'63 2024-04-25T07:16:32.513247+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:07.940573+0000 1 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-04-25T07:18:43.716910+0000 34'63 87:149 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:34.662771+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-04-25T07:18:43.717307+0000 34'64 87:168 [2,0,1] 2 [2,0,1] 2 34'64 2024-04-25T07:16:30.623123+0000 34'64 2024-04-25T07:16:30.623123+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:54:56.904166+0000 2 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:43.716033+0000 32'62 87:143 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:32.665083+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:18:43.715899+0000 36'63 87:140 [1,0,2] 1 [1,0,2] 1 32'62 2024-04-25T07:16:24.480791+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:06:26.260806+0000 2 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.683845+0000 34'64 87:175 [2,1,0] 2 [2,1,0] 2 34'64 2024-04-25T07:16:34.449560+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:35:05.986933+0000 3 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.e 4 1 1 0 0 2780 0 0 17 48 17 active+recovery_wait+degraded 2024-04-25T07:18:43.682851+0000 75'65 87:164 [1,2,0] 1 [1,2,0] 1 34'64 2024-04-25T07:16:38.451248+0000 34'64 2024-04-25T07:16:38.451248+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.f 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:43.687953+0000 34'63 87:160 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:38.567657+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-04-25T07:18:43.689311+0000 36'63 87:155 [0,2,1] 0 [0,2,1] 0 36'63 2024-04-25T07:16:33.620837+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:18:48.781063+0000 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.11 2 0 1 0 0 1195 0 0 14 48 14 active+recovering+degraded 2024-04-25T07:18:43.720802+0000 32'62 87:137 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:36.379171+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:01.569 INFO:tasks.workunit.client.0.smithi031.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-04-25T07:18:43.715551+0000 32'62 87:161 [0,1,2] 0 [0,1,2] 0 32'62 2024-04-25T07:16:34.551848+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:08:37.757307+0000 1 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-04-25T07:18:43.722391+0000 36'64 87:158 [1,2,0] 1 [1,2,0] 1 36'64 2024-04-25T07:16:42.545682+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.682010+0000 37'64 87:144 [1,0,2] 1 [1,0,2] 1 34'63 2024-04-25T07:16:29.519464+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:44:49.281033+0000 2 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-04-25T07:18:43.654685+0000 34'63 87:173 [2,0,1] 2 [2,0,1] 2 34'63 2024-04-25T07:16:37.422778+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:25:49.398896+0000 3 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-04-25T07:18:43.682956+0000 34'64 87:150 [1,0,2] 1 [1,0,2] 1 34'64 2024-04-25T07:16:44.542214+0000 34'64 2024-04-25T07:16:44.542214+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:44:45.271403+0000 4 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:43.714974+0000 32'62 87:149 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:41.566300+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-04-25T07:18:43.720313+0000 38'64 87:175 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:31.673045+0000 34'63 2024-04-25T07:16:31.673045+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:35:04.052387+0000 2 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-04-25T07:18:43.720248+0000 32'62 87:143 [1,2,0] 1 [1,2,0] 1 32'62 2024-04-25T07:16:49.496906+0000 0'0 2024-04-25T07:15:58.847017+0000 0 0 queued for scrub 0 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-04-25T07:18:43.716312+0000 34'63 87:148 [1,2,0] 1 [1,2,0] 1 34'63 2024-04-25T07:16:51.503641+0000 0'0 2024-04-25T07:15:58.847017+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:22:28.266260+0000 4 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:1.1b 4 1 1 0 0 2429 0 0 16 48 16 active+recovery_wait+degraded 2024-04-25T07:18:43.715516+0000 38'64 87:163 [1,0,2] 1 [1,0,2] 1 38'64 2024-04-25T07:16:37.721962+0000 38'64 2024-04-25T07:16:37.721962+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:01.570 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:19:01.585 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-04-25T07:19:01.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-04-25T07:19:01.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:19:01.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:19:01.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:19:01.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:19:01.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:19:01.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:19:01.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:19:01.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:19:01.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-04-25T07:19:01.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:19:01.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:19:01.757 INFO:tasks.workunit.client.0.smithi031.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:19:01.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:19:01.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:19:01.792 INFO:tasks.workunit.client.0.smithi031.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:19:01.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-04-25T07:19:01.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:19:01.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:19:01.814 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 42'65 (31'1) modify 1:d858c035:::existing_1:head by client.4633.0:1 2024-04-25T07:16:41.283842+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:19:01.814 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 20 _merge_object_divergent_entries: keeping 42'65 (31'1) modify 1:d858c035:::existing_1:head by client.4633.0:1 2024-04-25T07:16:41.283842+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:19:01.814 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 42'65 last_divergent_update: 42'65 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 42'66 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 42'65 (31'1) modify 1:d858c035:::existing_1:head by client.4633.0:1 2024-04-25T07:16:41.283842+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'48 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 42'66 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4634.0:1 2024-04-25T07:16:41.284939+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 20 _merge_object_divergent_entries: keeping 42'66 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4634.0:1 2024-04-25T07:16:41.284939+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 42'66 last_divergent_update: 42'66 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.674+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:19:01.815 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 42'65 (31'2) modify 1:97db2a0f:::existing_2:head by client.4637.0:1 2024-04-25T07:16:41.288035+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 20 _merge_object_divergent_entries: keeping 42'65 (31'2) modify 1:97db2a0f:::existing_2:head by client.4637.0:1 2024-04-25T07:16:41.288035+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 42'65 last_divergent_update: 42'65 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 42'65 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 42'65 (31'2) modify 1:97db2a0f:::existing_2:head by client.4637.0:1 2024-04-25T07:16:41.288035+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'48 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 42'66 (31'4) delete 1:70b3bf12:::existing_4:head by client.4636.0:1 2024-04-25T07:16:41.285631+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 20 _merge_object_divergent_entries: keeping 42'66 (31'4) delete 1:70b3bf12:::existing_4:head by client.4636.0:1 2024-04-25T07:16:41.285631+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 42'66 last_divergent_update: 42'66 2024-04-25T07:19:01.816 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: more recent entry found: 75'65 (31'4) modify 1:70b3bf12:::existing_4:head by client.4752.0:1 2024-04-25T07:17:45.489614+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:19:01.817 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 42'65 (0'0) modify 1:75ac5091:::newobject_2:head by client.4635.0:1 2024-04-25T07:16:41.284756+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:19:01.817 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 20 _merge_object_divergent_entries: keeping 42'65 (0'0) modify 1:75ac5091:::newobject_2:head by client.4635.0:1 2024-04-25T07:16:41.284756+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:19:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:19:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 42'65 last_divergent_update: 42'65 2024-04-25T07:19:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:19:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:31.770+0000 7ff299144640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:19:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 42'63 (31'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-04-25T07:16:41.282736+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.824 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 20 _merge_object_divergent_entries: keeping 42'63 (31'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-04-25T07:16:41.282736+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 42'63 last_divergent_update: 42'63 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 42'63 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 42'63 (31'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-04-25T07:16:41.282736+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:18:32.374+0000 7ff29a146640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'48 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 42'63 (31'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-04-25T07:16:41.282736+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 20 _merge_object_divergent_entries: keeping 42'63 (31'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-04-25T07:16:41.282736+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:19:01.825 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 42'63 last_divergent_update: 42'63 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 42'63 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 42'63 (31'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-04-25T07:16:41.282736+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:18:31.674+0000 7fb75d458640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'48 2024-04-25T07:19:01.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:19:01.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:19:01.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:19:01.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:19:01.863 INFO:tasks.workunit.client.0.smithi031.stdout:success 2024-04-25T07:19:01.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-04-25T07:19:01.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.25959.out 2024-04-25T07:19:01.864 INFO:tasks.workunit.client.0.smithi031.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.25959.out': No such file or directory 2024-04-25T07:19:01.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-04-25T07:19:01.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:19:01.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:19:02.385 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T07:19:02.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-04-25T07:19:02.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:19:02.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:19:02.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:19:02.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:19:02.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:19:03.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:19:03.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:19:03.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:19:03.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:19:03.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:19:03.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:19:03.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:19:03.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:19:03.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:19:03.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:19:03.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:19:03.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:19:03.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:19:03.731 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:19:03.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:19:03.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:19:03.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:19:03.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:19:03.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:19:03.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:19:03.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:19:03.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:19:03.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:19:03.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:19:03.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:19:03.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:19:03.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:19:03.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:19:03.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:19:03.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:19:03.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:19:03.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:19:03.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:19:03.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:19:03.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:19:03.781 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:19:03.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:19:03.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:19:03.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:19:03.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:19:03.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:03.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:03.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:19:03.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:19:03.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:19:03.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:19:03.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:19:03.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:03.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:03.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.25959 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-04-25T07:19:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-04-25T07:19:03.790 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-04-25T07:19:03.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-04-25T07:19:03.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:19:03.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:19:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:19:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:04.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:04.251 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:04.251 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:04.251 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:04.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:19:04.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:19:04.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:19:04.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:19:04.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:19:04.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:19:04.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:19:04.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:19:04.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get fsid 2024-04-25T07:19:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:19:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:19:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:19:04.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:19:04.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-mon.a.asok 2024-04-25T07:19:04.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:19:04.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.25959/ceph-mon.a.asok config get mon_host 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:19:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:19:05.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:19:05.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:05.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:05.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:05.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:05.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:05.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:19:05.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:19:05.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:19:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:19:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-04-25T07:19:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:19:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:19:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:19:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:05.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:19:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:19:05.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:19:05.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:19:05.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e7399d45-bbbb-4451-b676-87112ca8a79b' 2024-04-25T07:19:05.305 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:19:05.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:19:05.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpAypmLsfQExAAzv1B80qEVBcX9HgvIL6cOQ== 2024-04-25T07:19:05.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpAypmLsfQExAAzv1B80qEVBcX9HgvIL6cOQ=="}' 2024-04-25T07:19:05.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7399d45-bbbb-4451-b676-87112ca8a79b -i td/divergent-priors/0/new.json 2024-04-25T07:19:05.631 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:19:05.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:19:05.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpAypmLsfQExAAzv1B80qEVBcX9HgvIL6cOQ== --osd-uuid e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:19:05.683 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.682+0000 7f47a92597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:05.699 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.702+0000 7f47a92597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:05.701 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.702+0000 7f47a92597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:05.701 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.702+0000 7f47a92597c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:19:05.701 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.702+0000 7f47a92597c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:19:05.701 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.702+0000 7f47a92597c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:19:05.739 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:05.742+0000 7f47a92597c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:19:11.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:19:11.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:19:11.904 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:19:11.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:19:11.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:19:12.501 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:19:12.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:19:12.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:19:12.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:19:12.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:19:12.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:19:12.543 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:12.545+0000 7f414c6057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:12.562 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:12.561+0000 7f414c6057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:12.564 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:12.565+0000 7f414c6057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:12.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:19:12.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:19:12.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:19:12.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:19:12.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:19:12.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:12.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:19:12.969 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:19:12.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:12.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:13.112 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:13.113+0000 7f414c6057c0 -1 Falling back to public interface 2024-04-25T07:19:13.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:14.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:14.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:14.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:19:14.445 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:19:14.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:14.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:14.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:15.628 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:15.629+0000 7f414c6057c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:19:15.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:15.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:15.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:19:15.928 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:19:15.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:15.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:16.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:17.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:17.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:17.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:19:17.387 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:19:17.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:17.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:17.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:18.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:18.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:18.842 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:19:18.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:19:18.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:18.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:19.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:20.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:20.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:20.416 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:19:20.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:19:20.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:20.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:20.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:21.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:21.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:21.921 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:19:21.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:19:21.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:21.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:22.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:23.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:23.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:23.482 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:19:23.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:19:23.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:23.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:23.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:24.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:24.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:24.935 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:19:24.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:19:24.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:24.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:25.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:26.489 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:19:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:19:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:26.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:27.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:27.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:27.943 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:19:27.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:19:27.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:27.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:28.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:29.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:29.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:29.502 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:19:29.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:19:29.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:29.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:30.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:31.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:31.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:31.293 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:19:31.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:19:31.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:31.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:32.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:33.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:33.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:33.287 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:19:33.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:19:33.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:33.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:34.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:35.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:35.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:35.111 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:19:35.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:19:35.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:35.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:35.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:36.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:36.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:36.871 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:19:36.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:19:36.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:36.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:38.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:38.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:38.568 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:19:38.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:19:38.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:38.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:19:39.123 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/586482549,v1:127.0.0.1:6803/586482549] [v2:127.0.0.1:6804/586482549,v1:127.0.0.1:6805/586482549] exists,up e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:19:39.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:39.127 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:19:39.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:19:39.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:19:39.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:19:39.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39207523-3cd8-471c-b010-5a75fbb25cb6 2024-04-25T07:19:39.130 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 39207523-3cd8-471c-b010-5a75fbb25cb6 2024-04-25T07:19:39.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 39207523-3cd8-471c-b010-5a75fbb25cb6' 2024-04-25T07:19:39.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:19:39.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALBCpmX7RbCRAAeJgJepjTaXpx0tLpYoRJdA== 2024-04-25T07:19:39.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALBCpmX7RbCRAAeJgJepjTaXpx0tLpYoRJdA=="}' 2024-04-25T07:19:39.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39207523-3cd8-471c-b010-5a75fbb25cb6 -i td/divergent-priors/1/new.json 2024-04-25T07:19:39.672 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:19:39.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:19:39.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALBCpmX7RbCRAAeJgJepjTaXpx0tLpYoRJdA== --osd-uuid 39207523-3cd8-471c-b010-5a75fbb25cb6 2024-04-25T07:19:39.741 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.741+0000 7f6552f227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:39.752 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.753+0000 7f6552f227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:39.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.753+0000 7f6552f227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:39.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.753+0000 7f6552f227c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:19:39.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.753+0000 7f6552f227c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:19:39.755 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.753+0000 7f6552f227c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:19:39.844 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:39.845+0000 7f6552f227c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:19:45.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:19:45.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:19:45.871 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:19:45.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:19:45.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:19:46.478 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:19:46.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:19:46.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:19:46.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:19:46.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:19:46.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:19:46.522 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:46.521+0000 7f1a629397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:46.524 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:46.525+0000 7f1a629397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:46.526 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:46.525+0000 7f1a629397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:46.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:19:46.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:19:46.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:19:46.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:19:46.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:19:46.912 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:19:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:19:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:47.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:47.101+0000 7f1a629397c0 -1 Falling back to public interface 2024-04-25T07:19:47.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:48.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:48.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:48.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:19:48.333 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:19:48.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:48.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:48.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:49.647 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:19:49.649+0000 7f1a629397c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:19:49.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:49.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:49.806 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:19:49.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:19:49.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:49.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:50.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:51.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:51.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:51.391 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:19:51.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:19:51.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:51.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:51.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:52.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:52.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:52.847 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:19:52.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:19:52.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:52.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:53.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:54.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:54.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:54.474 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:19:54.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:19:54.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:54.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:54.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:55.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:55.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:55.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:19:55.928 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:19:55.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:55.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:56.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:57.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:57.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:57.504 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:19:57.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:19:57.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:57.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:57.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:58.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:58.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:58.966 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:19:58.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:19:58.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:58.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:59.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:00.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:00.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:00.535 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:20:00.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:20:00.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:00.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:00.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:01.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:01.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:01.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:20:01.985 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:20:01.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:01.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:02.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:03.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:03.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:03.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:20:03.795 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:20:03.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:03.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:04.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:05.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:05.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:05.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:20:05.315 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:20:05.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:05.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:06.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:07.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:07.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:07.168 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:20:07.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:20:07.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:07.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:07.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:08.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:08.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:08.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:20:08.941 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:20:08.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:08.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:10.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:11.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:11.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:11.172 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:20:11.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:20:11.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:11.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:12.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:13.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:13.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:13.197 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:20:13.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:20:13.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:13.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:13.689 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2378946785,v1:127.0.0.1:6811/2378946785] [v2:127.0.0.1:6812/2378946785,v1:127.0.0.1:6813/2378946785] exists,up 39207523-3cd8-471c-b010-5a75fbb25cb6 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:20:13.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:20:13.694 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:20:13.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:20:13.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:20:13.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:20:13.697 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:20:13.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8700d637-8430-4fba-817a-dba36e87e380' 2024-04-25T07:20:13.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:20:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtBCpmUnJxKxAAYEjWJ3OIwEN5GpkIPIZFZg== 2024-04-25T07:20:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtBCpmUnJxKxAAYEjWJ3OIwEN5GpkIPIZFZg=="}' 2024-04-25T07:20:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8700d637-8430-4fba-817a-dba36e87e380 -i td/divergent-priors/2/new.json 2024-04-25T07:20:14.325 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:20:14.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:20:14.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtBCpmUnJxKxAAYEjWJ3OIwEN5GpkIPIZFZg== --osd-uuid 8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:20:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.389+0000 7fd3c65627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:14.414 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.413+0000 7fd3c65627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:14.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.417+0000 7fd3c65627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:14.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.417+0000 7fd3c65627c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:20:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.417+0000 7fd3c65627c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:20:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.417+0000 7fd3c65627c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:20:14.505 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:14.505+0000 7fd3c65627c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:20:20.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:20:20.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:20:20.293 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:20:20.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:20:20.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:20:20.917 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:20:20.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:20:20.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:20.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:20:20.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:20:20.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:20:20.959 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:20.961+0000 7f0c3b5f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:20.961 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:20.961+0000 7f0c3b5f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:20.962 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:20.961+0000 7f0c3b5f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:21.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:20:21.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:21.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:20:21.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:21.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:21.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:21.394 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:20:21.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:21.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:21.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:21.505 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:21.505+0000 7f0c3b5f07c0 -1 Falling back to public interface 2024-04-25T07:20:21.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:22.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:22.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:22.859 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:20:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:23.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:24.071 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:20:24.073+0000 7f0c3b5f07c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:20:24.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:24.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:24.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:24.344 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:20:24.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:24.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:24.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:25.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:25.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:25.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:25.855 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:20:25.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:25.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:26.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:27.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:27.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:27.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:20:27.319 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:20:27.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:27.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:27.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:20:28.911 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:20:28.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:28.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:29.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:30.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:30.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:30.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:20:30.391 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:20:30.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:30.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:31.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:32.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:32.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:32.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:20:32.114 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:20:32.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:32.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:32.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:33.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:33.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:33.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:20:33.656 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:20:33.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:33.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:34.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:35.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:35.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:35.296 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:20:35.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:20:35.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:35.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:35.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:20:36.864 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:20:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:37.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:38.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:38.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:38.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:20:38.550 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:20:38.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:38.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:39.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:40.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:40.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:40.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:20:40.289 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:20:40.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:40.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:41.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:42.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:42.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:42.060 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:20:42.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:20:42.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:42.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:43.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:44.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:44.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:44.244 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:20:44.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:20:44.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:44.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:45.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:46.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:46.301 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:20:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:20:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:46.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:46.780 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2049125260,v1:127.0.0.1:6819/2049125260] [v2:127.0.0.1:6820/2049125260,v1:127.0.0.1:6821/2049125260] exists,up 8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:20:46.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:46.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:46.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:46.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-04-25T07:20:47.698 INFO:tasks.workunit.client.0.smithi031.stderr:noout is set 2024-04-25T07:20:47.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-04-25T07:20:48.460 INFO:tasks.workunit.client.0.smithi031.stderr:noin is set 2024-04-25T07:20:48.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-04-25T07:20:49.314 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T07:20:49.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-04-25T07:20:49.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=test 2024-04-25T07:20:49.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-04-25T07:20:49.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-04-25T07:20:49.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-04-25T07:20:49.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-25T07:20:49.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-04-25T07:20:49.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T07:20:50.765 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T07:20:50.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:20:51.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-04-25T07:20:51.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-04-25T07:20:52.538 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 allow_ec_overwrites to true 2024-04-25T07:20:52.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-04-25T07:20:52.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:20:52.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:20:52.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:20:52.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:20:52.558 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:20:52.558 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:20:52.558 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:20:52.558 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:20:52.558 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:20:52.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:20:52.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:20:52.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:20:52.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:20:52.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:20:52.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:20:53.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:20:53.111 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:20:53.111 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:20:53.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:20:53.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:53.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:20:53.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-04-25T07:20:53.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-04-25T07:20:53.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-04-25T07:20:53.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:53.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:20:53.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:20:53.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:20:53.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672975' 2024-04-25T07:20:53.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:53.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:20:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509447 2024-04-25T07:20:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509447 2024-04-25T07:20:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672975 2-64424509447' 2024-04-25T07:20:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:53.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-04-25T07:20:53.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:53.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:20:53.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-04-25T07:20:53.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:53.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-04-25T07:20:53.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-25T07:20:53.597 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836505 2024-04-25T07:20:53.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:20:54.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836505 2024-04-25T07:20:54.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:20:55.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:20:55.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:20:55.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-25T07:20:55.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:55.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:20:55.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:55.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:20:55.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:20:55.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:55.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:20:55.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:20:55.504 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:20:55.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:20:55.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-25T07:20:55.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:55.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:55.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509447 2024-04-25T07:20:55.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:20:55.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:55.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509447 2024-04-25T07:20:55.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509447 2024-04-25T07:20:55.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-04-25T07:20:55.936 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509447 2024-04-25T07:20:55.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:20:56.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509447 2024-04-25T07:20:56.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:20:56.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:56.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:56.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:20:56.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:56.940 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:56.940 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:56.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:56.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:56.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:56.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:57.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:20:57.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:57.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:57.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:57.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:20:57.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:20:57.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:20:57.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-04-25T07:20:57.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-04-25T07:20:57.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:20:57.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:20:58.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:20:58.339 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:20:58.339 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:20:58.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:20:58.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:58.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:20:58.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T07:20:58.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T07:20:58.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T07:20:58.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:58.523 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:20:58.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T07:20:58.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T07:20:58.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672978' 2024-04-25T07:20:58.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:58.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:20:58.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-04-25T07:20:58.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-04-25T07:20:58.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672978 2-64424509450' 2024-04-25T07:20:58.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:58.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T07:20:58.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:58.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:20:58.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T07:20:58.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:58.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T07:20:58.852 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836508 2024-04-25T07:20:58.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T07:20:58.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:20:59.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-04-25T07:20:59.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:21:00.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:21:00.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:21:00.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836508 2024-04-25T07:21:00.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:00.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T07:21:00.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:00.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:21:00.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T07:21:00.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:00.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T07:21:00.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T07:21:00.821 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672978 2024-04-25T07:21:00.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:21:01.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672978 2024-04-25T07:21:01.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:01.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:01.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-04-25T07:21:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:21:01.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-04-25T07:21:01.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:01.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-04-25T07:21:01.322 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509450 2024-04-25T07:21:01.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-25T07:21:01.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:21:01.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-25T07:21:01.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-04-25T07:21:01.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:21:01.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:21:01.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:21:01.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:21:01.803 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:21:01.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:21:01.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:21:01.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:21:01.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:21:01.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:21:01.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:21:01.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:21:01.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:21:01.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:21:01.925 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:21:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:21:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:21:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:21:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:21:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:21:02.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:21:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T07:21:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T07:21:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T07:21:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:21:02.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:21:02.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-04-25T07:21:02.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-04-25T07:21:02.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672981' 2024-04-25T07:21:02.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:21:02.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:21:02.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-04-25T07:21:02.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-04-25T07:21:02.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672981 2-64424509453' 2024-04-25T07:21:02.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:02.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T07:21:02.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:02.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:21:02.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T07:21:02.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:02.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T07:21:02.863 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836511 2024-04-25T07:21:02.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T07:21:02.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:21:03.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T07:21:03.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:21:04.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:21:04.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:21:04.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T07:21:04.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:04.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-04-25T07:21:04.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:04.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:21:04.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-04-25T07:21:04.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:04.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-04-25T07:21:04.827 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672981 2024-04-25T07:21:04.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-25T07:21:04.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:21:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-04-25T07:21:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:05.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-04-25T07:21:05.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:05.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:21:05.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-04-25T07:21:05.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:05.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-04-25T07:21:05.265 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509453 2024-04-25T07:21:05.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-25T07:21:05.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:21:05.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509453 2024-04-25T07:21:05.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:21:05.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:05.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:06.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:21:06.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:06.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:06.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:06.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:06.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:06.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:06.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:06.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:21:06.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:06.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:06.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:07.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:21:07.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:21:07.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:21:07.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-04-25T07:21:07.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-04-25T07:21:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:21:07.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-04-25T07:21:07.819 INFO:tasks.workunit.client.0.smithi031.stdout:primary and soon to be divergent is 1 2024-04-25T07:21:07.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-04-25T07:21:07.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:21:08.219 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:08.219 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:20:51.515594+0000 0'0 29:15 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:01:13.821499+0000 0 0 2024-04-25T07:21:08.220 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:21:08.220 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:08.220 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:21:08.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-04-25T07:21:08.236 INFO:tasks.workunit.client.0.smithi031.stdout:writing initial objects 2024-04-25T07:21:08.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-04-25T07:21:08.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:08.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-04-25T07:21:08.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:08.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-04-25T07:21:08.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:08.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-04-25T07:21:08.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:08.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-04-25T07:21:09.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:09.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-04-25T07:21:09.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:09.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-04-25T07:21:09.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:09.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-04-25T07:21:09.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:09.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-04-25T07:21:09.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:09.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-04-25T07:21:10.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:10.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-04-25T07:21:10.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:10.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-04-25T07:21:10.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:10.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-04-25T07:21:10.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:10.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-04-25T07:21:10.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:10.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-04-25T07:21:11.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:11.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-04-25T07:21:11.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:11.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-04-25T07:21:11.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:11.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-04-25T07:21:11.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:11.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-04-25T07:21:11.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:11.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-04-25T07:21:12.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:12.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-04-25T07:21:12.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:12.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-04-25T07:21:12.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:12.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-04-25T07:21:12.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:12.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-04-25T07:21:12.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:12.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-04-25T07:21:13.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:13.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-04-25T07:21:13.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:13.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-04-25T07:21:13.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:13.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-04-25T07:21:13.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:13.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-04-25T07:21:14.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:14.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-04-25T07:21:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-04-25T07:21:14.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:14.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-04-25T07:21:14.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:14.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-04-25T07:21:14.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:14.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-04-25T07:21:15.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:15.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-04-25T07:21:15.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:15.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-04-25T07:21:15.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:15.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-04-25T07:21:15.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:15.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-04-25T07:21:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-04-25T07:21:16.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:16.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-04-25T07:21:16.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:16.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-04-25T07:21:16.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:16.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-04-25T07:21:16.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:16.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-04-25T07:21:17.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:17.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-04-25T07:21:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-04-25T07:21:17.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:17.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-04-25T07:21:17.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:17.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-04-25T07:21:17.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:17.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-04-25T07:21:18.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:18.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-04-25T07:21:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-04-25T07:21:18.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:18.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-04-25T07:21:18.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:18.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-04-25T07:21:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-04-25T07:21:19.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:19.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-04-25T07:21:19.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:19.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-04-25T07:21:19.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:19.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-04-25T07:21:19.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:19.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-04-25T07:21:19.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:19.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-04-25T07:21:20.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:20.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-04-25T07:21:20.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:20.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-04-25T07:21:20.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:20.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-04-25T07:21:20.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:20.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-04-25T07:21:21.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:21.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-04-25T07:21:21.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:21.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-04-25T07:21:21.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:21.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-04-25T07:21:21.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:21.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-04-25T07:21:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-04-25T07:21:22.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:22.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-04-25T07:21:22.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:22.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-04-25T07:21:22.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:22.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-04-25T07:21:22.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:22.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-04-25T07:21:22.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:22.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-04-25T07:21:23.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:23.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-04-25T07:21:23.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:23.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-04-25T07:21:23.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:23.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-04-25T07:21:23.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:23.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-04-25T07:21:23.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:23.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-04-25T07:21:24.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:24.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-04-25T07:21:24.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:24.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-04-25T07:21:24.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:24.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-04-25T07:21:24.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:24.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-04-25T07:21:25.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:25.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-04-25T07:21:25.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:25.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-04-25T07:21:25.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:25.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-04-25T07:21:25.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:25.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-04-25T07:21:25.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:25.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-04-25T07:21:26.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:26.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-04-25T07:21:26.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:26.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-04-25T07:21:26.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:26.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-04-25T07:21:26.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:26.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-04-25T07:21:26.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:26.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-04-25T07:21:27.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:27.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-04-25T07:21:27.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:27.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-04-25T07:21:27.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:27.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-04-25T07:21:27.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:27.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-04-25T07:21:27.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:27.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-04-25T07:21:27.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:27.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-04-25T07:21:28.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:28.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-04-25T07:21:28.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:28.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-04-25T07:21:28.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:28.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:21:28.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:21:28.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:21:28.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:21:28.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-04-25T07:21:28.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:21:28.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:21:28.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:21:28.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:21:28.967 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:21:28.967 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:21:28.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:21:28.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:21:28.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:21:29.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:21:29.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:21:29.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:21:29.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:21:29.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:21:29.050 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:21:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:21:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:21:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:21:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:21:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:21:29.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:21:29.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836519 2024-04-25T07:21:29.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836519 2024-04-25T07:21:29.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836519' 2024-04-25T07:21:29.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:21:29.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:21:29.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-04-25T07:21:29.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-04-25T07:21:29.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836519 1-42949672989' 2024-04-25T07:21:29.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:21:29.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:21:30.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-04-25T07:21:30.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-04-25T07:21:30.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836519 1-42949672989 2-64424509460' 2024-04-25T07:21:30.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:30.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836519 2024-04-25T07:21:30.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:30.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:21:30.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836519 2024-04-25T07:21:30.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:30.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836519 2024-04-25T07:21:30.044 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836519 2024-04-25T07:21:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-04-25T07:21:30.044 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:21:30.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836519 -lt 21474836519 2024-04-25T07:21:30.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:30.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-04-25T07:21:30.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:30.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:21:30.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-04-25T07:21:30.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-04-25T07:21:30.489 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672989 2024-04-25T07:21:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-25T07:21:30.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:21:30.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-25T07:21:30.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:21:30.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-04-25T07:21:30.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:21:30.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:21:30.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-04-25T07:21:30.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:21:30.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-04-25T07:21:30.932 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509460 2024-04-25T07:21:30.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-25T07:21:30.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:21:31.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-04-25T07:21:31.471 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:21:31.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:31.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:32.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:21:32.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:32.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:32.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:32.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:32.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:32.048 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:32.048 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:32.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:21:32.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:32.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:32.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:21:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:21:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:21:33.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-04-25T07:21:33.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:21:33.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:21:33.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:21:33.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:21:33.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-04-25T07:21:33.529 INFO:tasks.workunit.client.0.smithi031.stdout:blackholing osds 0 2 2024-04-25T07:21:33.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-04-25T07:21:33.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:21:33.933 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:33.933 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:20:51.515594+0000 29'100 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:01:13.821499+0000 0 0 2024-04-25T07:21:33.933 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:21:33.933 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:33.934 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:21:33.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:33.951 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:33.951 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:21:33.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:21:33.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:21:33.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:21:34.062 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:21:34.062 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:21:34.062 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:21:34.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:34.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:21:34.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:21:34.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:21:34.077 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:34.077 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:34.077 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:21:34.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:21:34.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:21:34.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:21:34.184 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:21:34.184 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:21:34.184 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:21:34.197 INFO:tasks.workunit.client.0.smithi031.stdout:writing divergent object 2024-04-25T07:21:34.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-04-25T07:21:34.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-04-25T07:21:34.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-04-25T07:21:35.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-04-25T07:21:35.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:21:35.409 INFO:tasks.workunit.client.0.smithi031.stdout:created pool test snap snap1 2024-04-25T07:21:35.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-04-25T07:21:35.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-04-25T07:21:35.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:21:45.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-04-25T07:21:45.421 INFO:tasks.workunit.client.0.smithi031.stderr:rados: no process found 2024-04-25T07:21:45.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-04-25T07:21:45.422 INFO:tasks.workunit.client.0.smithi031.stdout:killing all the osds 2024-04-25T07:21:45.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:21:45.842 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:45.843 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-04-25T07:20:51.515594+0000 30'104 30:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:20:34.709262+0000 0 0 2024-04-25T07:21:45.843 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:21:45.843 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:45.843 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:21:45.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:21:45.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:21:45.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:21:45.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:21:45.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:21:45.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:21:45.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:21:45.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:45.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-04-25T07:21:46.509 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:21:46.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:46.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-04-25T07:21:47.073 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:21:47.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:47.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-04-25T07:21:47.649 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:21:47.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:47.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-04-25T07:21:48.208 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T07:21:48.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:48.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-04-25T07:21:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already out. 2024-04-25T07:21:48.858 INFO:tasks.workunit.client.0.smithi031.stdout:bringing up non_divergent 0 2 2024-04-25T07:21:48.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-04-25T07:21:48.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:21:49.272 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:49.273 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-04-25T07:20:51.515594+0000 30'104 30:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:20:34.709262+0000 0 0 2024-04-25T07:21:49.273 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:21:49.273 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:49.273 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:21:49.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:49.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:21:49.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:21:49.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:21:49.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:21:49.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:21:49.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:21:49.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:49.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:49.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:21:49.294 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:21:49.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:21:49.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:49.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:21:49.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:21:49.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:21:49.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:21:49.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:21:49.335 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:49.340+0000 7f9b009ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:49.338 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:49.340+0000 7f9b009ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:49.339 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:49.344+0000 7f9b009ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:49.739 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:21:49.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:49.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:49.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:49.900 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:49.904+0000 7f9b009ea7c0 -1 Falling back to public interface 2024-04-25T07:21:50.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:51.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:51.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:51.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:21:51.190 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:21:51.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:51.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:51.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:52.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:52.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:52.638 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:21:52.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:21:52.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:52.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:52.968 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:52.972+0000 7f9b009ea7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:21:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:54.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:54.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:54.134 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:21:54.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:21:54.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:54.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:54.595 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1856252473,v1:127.0.0.1:6803/1856252473] [v2:127.0.0.1:6804/1856252473,v1:127.0.0.1:6805/1856252473] exists,up e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:21:54.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:54.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:54.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:54.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:54.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:21:54.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:54.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:21:54.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:21:54.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:21:54.602 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:21:54.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:21:54.634 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:54.640+0000 7fbdb97be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:54.636 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:54.640+0000 7fbdb97be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:54.638 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:54.640+0000 7fbdb97be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:55.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:21:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:21:55.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:55.196+0000 7fbdb97be7c0 -1 Falling back to public interface 2024-04-25T07:21:55.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:56.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:56.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:56.564 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:21:56.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:21:56.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:56.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:21:57.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:58.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:58.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:58.031 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:21:58.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:21:58.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:58.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:21:58.040 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:21:58.044+0000 7fbdb97be7c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:21:58.607 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2067748886,v1:127.0.0.1:6811/2067748886] [v2:127.0.0.1:6812/2067748886,v1:127.0.0.1:6813/2067748886] exists,up 8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:21:58.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:58.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:58.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:58.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:58.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-04-25T07:21:59.187 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. 2024-04-25T07:21:59.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:21:59.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-04-25T07:21:59.797 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already in. 2024-04-25T07:21:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-04-25T07:22:04.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:22:04.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-04-25T07:22:04.819 INFO:tasks.workunit.client.0.smithi031.stdout:writing non-divergent object existing_4 2024-04-25T07:22:04.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-04-25T07:22:04.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:22:05.238 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:22:05.238 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-04-25T07:20:51.515594+0000 30'104 30:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:20:34.709262+0000 0 0 2024-04-25T07:22:05.239 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:22:05.239 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:22:05.239 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:22:05.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:22:05.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:22:05.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-04-25T07:22:05.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:22:05.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:22:05.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:22:05.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:22:05.355 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:22:05.355 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:22:05.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:22:05.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:22:05.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:22:05.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:22:05.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:22:05.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:22:05.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:22:05.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:22:05.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:22:05.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:22:05.867 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:22:05.867 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:22:05.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:22:05.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:22:05.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:22:06.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789957 2024-04-25T07:22:06.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789957 2024-04-25T07:22:06.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789957' 2024-04-25T07:22:06.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:22:06.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:22:06.162 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:22:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:22:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:22:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:22:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:22:06.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:22:06.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=167503724548 2024-04-25T07:22:06.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 167503724548 2024-04-25T07:22:06.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-04-25T07:22:06.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:22:06.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-158913789957 2024-04-25T07:22:06.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:22:06.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:22:06.332 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-158913789957 2024-04-25T07:22:06.332 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:22:06.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789957 2024-04-25T07:22:06.334 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 158913789957 2024-04-25T07:22:06.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-04-25T07:22:06.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:22:06.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789957 -lt 158913789957 2024-04-25T07:22:06.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:22:06.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-167503724548 2024-04-25T07:22:06.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:22:06.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:22:06.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-167503724548 2024-04-25T07:22:06.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:22:06.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=167503724548 2024-04-25T07:22:06.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-04-25T07:22:06.856 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 167503724548 2024-04-25T07:22:06.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:22:07.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724546 -lt 167503724548 2024-04-25T07:22:07.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:22:08.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:22:08.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:22:08.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724548 -lt 167503724548 2024-04-25T07:22:08.818 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:22:08.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:08.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:09.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:22:09.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:09.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:09.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:09.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:09.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:09.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:09.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:09.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:09.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:09.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:09.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:10.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:10.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:22:10.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:22:10.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:22:10.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:10.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:22:10.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:10.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:10.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:10.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:10.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:10.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:10.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:10.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:10.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:10.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:10.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:10.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:11.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:11.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:11.482 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:12.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:12.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:12.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:22:12.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:12.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:22:12.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:12.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:12.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:12.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:12.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:12.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:12.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:12.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:12.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:12.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:12.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:12.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:13.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:13.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:13.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:13.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:13.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:22:13.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:13.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:22:14.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:14.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:14.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:14.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:14.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:14.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:14.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:14.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:14.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:14.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:14.644 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:14.644 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:15.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:15.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:15.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:15.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:15.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:15.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:15.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:15.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:15.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:22:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:22:16.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:16.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:16.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:16.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:16.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:16.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:16.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:16.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:17.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:17.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:17.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:17.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:17.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:17.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:17.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:17.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:17.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:18.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:18.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:18.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:22:18.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:18.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:22:19.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:19.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:19.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:19.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:19.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:19.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:19.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:19.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:20.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:20.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:20.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:20.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:20.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:20.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:22:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:22:24.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:24.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:24.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:24.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:24.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:24.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:24.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:24.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:25.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:25.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:25.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:25.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:25.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:25.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:25.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:25.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:25.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:25.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:25.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:25.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:25.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:26.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:26.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:22:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:22:32.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:32.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:32.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:32.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:32.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:32.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:32.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:32.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:32.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:32.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:32.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:32.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:33.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:33.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:33.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:34.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:34.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:34.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:22:34.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:34.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:22:41.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:41.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:41.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:41.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:41.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:41.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:41.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:41.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:41.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:41.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:41.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:41.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:42.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:42.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:42.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:42.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:42.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:42.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:42.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:42.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:42.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:22:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "71e8cc48-2ec7-4fd8-9e20-29959d0adbaf", 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:22:43.577632+0000", 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:22:43.574 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:22:43.575 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:22:43.576 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.577 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.578 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:19:03.823579Z", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:19:03.823579Z", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:22:43.579 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.580 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:22:43.581 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 43, 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:19:04.749549+0000", 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:22:00.218466+0000", 2024-04-25T07:22:43.582 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:21:58.389962+0000", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:21:59.592722+0000", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.583 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 10, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:22:43.584 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:20:50.424736+0000", 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 16389, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 3, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 1, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 1, 2024-04-25T07:22:43.585 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 1, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 1, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 1, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 1, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "30", 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:22:43.586 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "pool", 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 1, 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 30, 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [ 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "snapid": 1, 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "stamp": "2024-04-25T07:21:35.248998+0000", 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "snap1" 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:22:43.587 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:22:43.588 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 8192, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:22:43.589 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {} 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "e7399d45-bbbb-4451-b676-87112ca8a79b", 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 5, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 37, 2024-04-25T07:22:43.590 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 41, 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 31, 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.591 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.592 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.593 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1856252473 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.594 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/1856252473", 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/1856252473", 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1856252473", 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1856252473", 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "39207523-3cd8-471c-b010-5a75fbb25cb6", 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:22:43.595 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 25, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 32, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.596 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.597 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.598 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.599 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/2378946785", 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/2378946785", 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2378946785", 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2378946785", 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.600 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "8700d637-8430-4fba-817a-dba36e87e380", 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 15, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 29, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 39, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 0, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 33, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.601 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.602 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.603 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.604 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2067748886 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/2067748886", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/2067748886", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2067748886", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2067748886", 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:22:43.605 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:21:46.353146+0000", 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:19:37.566273+0000", 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:21:46.912575+0000", 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:20:12.697822+0000", 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:22:43.607 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:21:47.475444+0000", 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:20:45.721011+0000", 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:22:43.608 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "myprofile": { 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-device-class": "", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-failure-domain": "osd", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-root": "default", 2024-04-25T07:22:43.609 INFO:tasks.workunit.client.0.smithi031.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "1", 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van", 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "w": "8" 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:22:43.610 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/1856252473,v1:127.0.0.1:6805/1856252473]", 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:22:43.611 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:22:43.612 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:19:11.192150Z", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/1856252473,v1:127.0.0.1:6803/1856252473]", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:22:43.613 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1856252473,v1:127.0.0.1:6809/1856252473]", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1856252473,v1:127.0.0.1:6807/1856252473]", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:22:43.614 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/2378946785,v1:127.0.0.1:6813/2378946785]", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:22:43.615 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:19:45.200391Z", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:22:43.616 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/2378946785,v1:127.0.0.1:6811/2378946785]", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2378946785,v1:127.0.0.1:6817/2378946785]", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2378946785,v1:127.0.0.1:6815/2378946785]", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:22:43.617 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/2067748886,v1:127.0.0.1:6813/2067748886]", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:22:43.618 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:22:43.619 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:20:19.622334Z", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:22:43.620 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/2067748886,v1:127.0.0.1:6811/2067748886]", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2067748886,v1:127.0.0.1:6817/2067748886]", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2067748886,v1:127.0.0.1:6815/2067748886]", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:22:43.621 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 37 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.622 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 39 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 43, 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.623 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.624 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.625 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:22:43.626 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:22:43.627 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:22:43.628 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:22:43.629 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:22:43.630 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.631 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:22:43.632 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:22:43.633 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.634 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 1, 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "test", 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 3, 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:22:43.635 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 5 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "set_choose_tries", 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 100 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_indep", 2024-04-25T07:22:43.636 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.637 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stderr:report 3981266219 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:22:43.638 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:22:43.639 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:22:43.640 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:22:43.641 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 1, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:22:43.642 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 62667, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 101, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 1, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 303, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 101, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 101, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 101, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 101, 2024-04-25T07:22:43.643 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:22:43.644 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:22:43.645 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 18, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 18, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 3, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 3, 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:22:43.646 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83672, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2164, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489128, 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:22:43.647 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036867072, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2215936, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 1550569, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416932 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:22:43.648 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:22:43.649 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 83672, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2164, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 10, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 81461, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314489128, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:22:43.650 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322036867072, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2215936, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 1550569, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 10396, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 83416932 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:22:43.651 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:22:43.652 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 62667, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 101, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 1, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 303, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 101, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 101, 2024-04-25T07:22:43.653 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 101, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 101, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:22:43.654 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:22:43.655 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1241088, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 1241088, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.656 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 18, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 18, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 3, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 3, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 158913789964 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672992 2024-04-25T07:22:43.657 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 167503724555 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 1 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:22:43.658 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2147483647, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:22:43.659 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:22:43.660 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 5, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:21:48.038577+0000", 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 98, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 397, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 544, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:22:43.661 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:22:43.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:22:43.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:22:43.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:22:43.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-04-25T07:22:43.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:22:43.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:22:43.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-25T07:22:43.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:22:43.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-04-25T07:22:43.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-04-25T07:22:44.919 INFO:tasks.workunit.client.0.smithi031.stderr:PG '1.0' not found 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:22:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:22:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:22:45.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:22:45.226 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:22:45.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:22:45.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:45.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:22:45.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:22:45.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:22:45.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:22:45.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:22:45.276 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:45.279+0000 7f03989da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:45.279 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:45.283+0000 7f03989da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:45.280 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:45.283+0000 7f03989da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:45.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:22:45.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:45.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:22:45.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:45.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:45.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:45.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:45.726 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:22:45.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:45.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:45.856 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:45.859+0000 7f03989da7c0 -1 Falling back to public interface 2024-04-25T07:22:46.212 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 37 up_thru 41 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1856252473,v1:127.0.0.1:6803/1856252473] [v2:127.0.0.1:6804/1856252473,v1:127.0.0.1:6805/1856252473] exists,up e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:22:46.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:46.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:46.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:46.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:22:46.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-04-25T07:22:46.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:22:46.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:22:46.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:22:46.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:22:46.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-04-25T07:22:46.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-04-25T07:22:47.391 INFO:tasks.workunit.client.0.smithi031.stderr:PG '1.0' not found 2024-04-25T07:22:47.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:22:47.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:22:47.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:22:47.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:22:47.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:22:47.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:47.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:47.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:22:47.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:22:47.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:22:47.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:22:47.934 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:22:47.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:47.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:22:47.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:22:47.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:22:47.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:22:47.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:22:47.980 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:47.983+0000 7f00769927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:47.983 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:47.987+0000 7f00769927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:47.985 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:47.987+0000 7f00769927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:48.384 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:22:48.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:48.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:22:48.518 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:48.523+0000 7f03989da7c0 -1 osd.0 43 log_to_monitors true 2024-04-25T07:22:48.547 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:48.551+0000 7f00769927c0 -1 Falling back to public interface 2024-04-25T07:22:48.930 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2067748886,v1:127.0.0.1:6811/2067748886] [v2:127.0.0.1:6812/2067748886,v1:127.0.0.1:6813/2067748886] exists,up 8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:22:48.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-04-25T07:22:50.569 INFO:tasks.workunit.client.0.smithi031.stderr:PG '1.0' not found 2024-04-25T07:22:50.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:22:50.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:22:50.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-04-25T07:22:50.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:22:50.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:22:50.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:22:50.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:22:50.865 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:22:50.865 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:22:50.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:22:50.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:22:50.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:22:50.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:22:50.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:22:50.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:22:50.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:22:50.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:22:50.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:22:51.006 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:22:51.007+0000 7f00769927c0 -1 osd.2 43 log_to_monitors true 2024-04-25T07:22:51.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:22:51.468 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:22:51.468 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:22:51.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:22:51.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:22:51.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:22:51.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495619 2024-04-25T07:22:51.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495619 2024-04-25T07:22:51.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619' 2024-04-25T07:22:51.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:22:51.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:22:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:22:51.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:22:51.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:22:51.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:22:51.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:22:51.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:22:52.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364803 2024-04-25T07:22:52.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364803 2024-04-25T07:22:52.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-04-25T07:22:52.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:22:52.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-197568495619 2024-04-25T07:22:52.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:22:52.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:22:52.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-197568495619 2024-04-25T07:22:52.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:22:52.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495619 2024-04-25T07:22:52.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-04-25T07:22:52.012 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 197568495619 2024-04-25T07:22:52.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:22:52.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495619 -lt 197568495619 2024-04-25T07:22:52.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:22:52.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364803 2024-04-25T07:22:52.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:22:52.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:22:52.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364803 2024-04-25T07:22:52.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:22:52.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364803 2024-04-25T07:22:52.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-04-25T07:22:52.529 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 214748364803 2024-04-25T07:22:52.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:22:52.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364803 -lt 214748364803 2024-04-25T07:22:52.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:22:52.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:52.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:53.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:22:53.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:53.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:53.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:53.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:53.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:53.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:53.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:53.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:53.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:53.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:53.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:54.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:54.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:22:54.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:22:54.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:22:54.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:54.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:22:54.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:54.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:54.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:54.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:54.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:54.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:54.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:54.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:55.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:55.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:55.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:55.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:55.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:55.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:55.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:55.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:55.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:55.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:55.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:55.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:55.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:56.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:56.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:56.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:22:56.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:56.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:22:56.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:56.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:56.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:56.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:56.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:56.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:56.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:56.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:56.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:56.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:56.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:56.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:57.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:57.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:57.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:22:58.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:22:58.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:22:58.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:22:58.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:22:58.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:22:58.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:22:58.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:22:58.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:22:58.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:22:58.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:22:58.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:22:58.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:22:58.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:22:58.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:22:58.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:22:58.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:22:58.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:22:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:22:59.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:22:59.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:23:00.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:23:00.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:23:00.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:23:00.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:23:00.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:23:00.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:23:00.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:00.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:00.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:00.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:00.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:00.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:00.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:01.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:23:01.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:01.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:01.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:23:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:23:01.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:23:01.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:23:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:23:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:23:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:23:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:23:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:23:03.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:23:03.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:03.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:03.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:03.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:03.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:03.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:03.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:04.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:23:04.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:23:05.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:23:05.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:23:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:23:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:23:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:23:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:23:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:23:08.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:23:08.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:08.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:08.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:09.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:23:09.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:09.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:09.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:23:09.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:23:10.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:23:10.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:23:10.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:23:10.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:23:10.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:23:16.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:23:16.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:16.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:16.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:16.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:16.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:16.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:16.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:17.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:23:17.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:17.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:17.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:23:17.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:23:17.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:23:17.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:23:18.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:23:18.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:23:18.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:23:18.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:23:18.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:23:25.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:23:25.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:25.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:25.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:25.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:25.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:25.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:25.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:26.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:23:26.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:26.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:26.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:23:26.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:23:26.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:23:26.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:23:27.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:23:27.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:23:27.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:23:27.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "71e8cc48-2ec7-4fd8-9e20-29959d0adbaf", 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T07:23:27.788167+0000", 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T07:23:27.785 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_WARN", 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 17 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T07:23:27.786 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "OSD_DOWN": { 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 osds down", 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "osd.1 (root=default,host=smithi031) is down" 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.787 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.788 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.789 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T07:23:27.790 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:19:03.823579Z", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:19:03.823579Z", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T07:23:27.791 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T07:23:27.792 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:23:27.793 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.794 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 52, 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "edeed592-e028-4157-aeea-12aafe002180", 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T07:19:04.749549+0000", 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:22:52.107154+0000", 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T07:22:51.478643+0000", 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T07:21:59.592722+0000", 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:23:27.795 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5800384, 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "nodown", 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "noin", 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "noout", 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 12, 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:23:27.796 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 1, 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 3, 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "jewel", 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "test", 2024-04-25T07:23:27.797 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T07:20:50.424736+0000", 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 16389, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 3, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 1, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:23:27.798 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 1, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 1, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 1, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 1, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 1, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T07:23:27.799 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "30", 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "pool", 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 1, 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 30, 2024-04-25T07:23:27.800 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [ 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "snapid": 1, 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "stamp": "2024-04-25T07:21:35.248998+0000", 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "snap1" 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T07:23:27.801 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T07:23:27.802 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 8192, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T07:23:27.803 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {} 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "e7399d45-bbbb-4451-b676-87112ca8a79b", 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:23:27.804 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 37, 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 43, 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 46, 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 51, 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 45, 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.805 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.806 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.807 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.808 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 191617378 2024-04-25T07:23:27.809 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/191617378", 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/191617378", 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/191617378", 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/191617378", 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.810 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "39207523-3cd8-471c-b010-5a75fbb25cb6", 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 0, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 10, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 25, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 32, 2024-04-25T07:23:27.811 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.812 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.813 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.814 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.815 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 2378946785 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/2378946785", 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/2378946785", 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2378946785", 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2378946785", 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "exists" 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.816 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "8700d637-8430-4fba-817a-dba36e87e380", 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 39, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 43, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 50, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 0, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 49, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.817 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T07:23:27.818 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.819 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.820 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T07:23:27.821 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3001253986 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/3001253986", 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/3001253986", 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3001253986", 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3001253986", 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T07:23:27.822 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:22:48.816466+0000", 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:23:27.823 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:19:37.566273+0000", 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:21:46.912575+0000", 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:20:12.697822+0000", 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.824 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "2024-04-25T07:22:51.328056+0000", 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T07:20:45.721011+0000", 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [], 2024-04-25T07:23:27.825 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.826 INFO:tasks.workunit.client.0.smithi031.stdout: "myprofile": { 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-device-class": "", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-failure-domain": "osd", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "crush-root": "default", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "1", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van", 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: "w": "8" 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.827 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T07:23:27.828 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/191617378,v1:127.0.0.1:6805/191617378]", 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:23:27.829 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:23:27.830 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:19:11.192150Z", 2024-04-25T07:23:27.831 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/191617378,v1:127.0.0.1:6803/191617378]", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/191617378,v1:127.0.0.1:6809/191617378]", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/191617378,v1:127.0.0.1:6807/191617378]", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:23:27.832 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:23:27.833 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/2378946785,v1:127.0.0.1:6813/2378946785]", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:23:27.834 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:23:27.835 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:19:45.200391Z", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/2378946785,v1:127.0.0.1:6811/2378946785]", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:23:27.836 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2378946785,v1:127.0.0.1:6817/2378946785]", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2378946785,v1:127.0.0.1:6815/2378946785]", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:23:27.837 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/3001253986,v1:127.0.0.1:6813/3001253986]", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:23:27.838 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:23:27.839 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T07:20:19.622334Z", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/3001253986,v1:127.0.0.1:6811/3001253986]", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T07:23:27.840 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3001253986,v1:127.0.0.1:6817/3001253986]", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3001253986,v1:127.0.0.1:6815/3001253986]", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:23:27.841 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 26, 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [ 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: "floor": 26 2024-04-25T07:23:27.842 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 46 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 50 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.843 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 52, 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:23:27.844 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.845 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.846 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.847 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T07:23:27.848 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.849 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.850 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.851 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:23:27.852 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.853 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T07:23:27.854 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 19206, 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:23:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T07:23:27.856 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:23:27.857 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 1, 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "test", 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 3, 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T07:23:27.858 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 5 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "set_choose_tries", 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 100 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.859 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_indep", 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:23:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T07:23:27.861 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 1, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T07:23:27.862 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T07:23:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.864 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 5, 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 3 2024-04-25T07:23:27.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 1, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 3, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 62667, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 101, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 1, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 303, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:23:27.866 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 101, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 101, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 101, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 101, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:23:27.867 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:23:27.868 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.870 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 18, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 18, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 3, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 3, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:23:27.871 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 85116, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2308, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 15, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 82736, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314487684, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322035388416, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2363392, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 1611753, 2024-04-25T07:23:27.872 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 16263, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 84721785 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:23:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T07:23:27.874 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 3, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 3, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 3, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 314572800, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 85116, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 2308, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 15, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 82736, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 314487684, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 322122547200, 2024-04-25T07:23:27.875 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 322035388416, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 2363392, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 1611753, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 16263, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 84721785 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T07:23:27.876 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 0, 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [], 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 1 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T07:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 62667, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 101, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 1, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 303, 2024-04-25T07:23:27.878 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 101, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 101, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 101, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 101, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:23:27.879 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T07:23:27.880 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1241088, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 1241088, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T07:23:27.882 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 18, 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 18, 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 3, 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 3, 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:23:27.883 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 197568495626 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 42949672992 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 214748364809 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.884 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+undersized+degraded", 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 1 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2147483647, 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T07:23:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 1, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 0, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 0, 2024-04-25T07:23:27.886 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 5, 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T07:21:48.038577+0000", 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T07:23:27.887 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 120, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 480, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 657, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T07:23:27.888 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:23:27.889 INFO:tasks.workunit.client.0.smithi031.stderr:report 233202824 2024-04-25T07:23:27.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:23:27.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-04-25T07:23:27.889 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery 2024-04-25T07:23:27.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:28.186 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:28.186 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:22:52.453633+0000 43'102 52:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:23:28.187 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:28.187 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:28.187 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:28.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:23:28.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:23:28.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:23:28.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:23:28.203 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:23:28.203 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:28.203 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:23:28.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.0.asok 2024-04-25T07:23:28.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:23:28.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:23:28.319 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:23:28.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:23:28.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:23:28.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:23:28.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:23:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:23:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:23:28.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.2.asok 2024-04-25T07:23:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:23:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:23:28.453 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:23:28.467 INFO:tasks.workunit.client.0.smithi031.stdout:revive divergent 1 2024-04-25T07:23:28.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-04-25T07:23:28.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:28.871 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:28.871 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:22:52.453633+0000 43'102 52:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:23:28.871 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:28.871 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:28.871 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:28.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-04-25T07:23:29.557 INFO:tasks.workunit.client.0.smithi031.stderr:noup is set 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:23:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:23:29.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:23:29.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:23:29.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:23:29.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:23:29.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:23:29.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:23:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:23:29.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:23:29.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:23:29.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:23:29.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:23:29.581 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:23:29.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:23:29.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:23:29.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:23:29.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:23:29.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:23:29.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:23:29.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:23:29.623 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:29.626+0000 7fc063f697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:29.625 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:29.626+0000 7fc063f697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:29.626 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:29.630+0000 7fc063f697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:30.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:23:30.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-04-25T07:23:30.197 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:30.198+0000 7fc063f697c0 -1 Falling back to public interface 2024-04-25T07:23:32.816 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:32.818+0000 7fc063f697c0 -1 osd.1 30 log_to_monitors true 2024-04-25T07:23:35.034 INFO:tasks.workunit.client.0.smithi031.stdout:delay recovery divergent 2024-04-25T07:23:35.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-04-25T07:23:35.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:35.468 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:35.468 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:22:52.453633+0000 43'102 55:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:01:11.123784+0000 0 0 2024-04-25T07:23:35.468 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:35.468 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:35.469 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:23:35.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25959/ceph-osd.1.asok 2024-04-25T07:23:35.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:23:35.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.25959/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:23:35.605 INFO:tasks.workunit.client.0.smithi031.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:23:35.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-04-25T07:23:36.310 INFO:tasks.workunit.client.0.smithi031.stderr:noup is unset 2024-04-25T07:23:36.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-04-25T07:23:36.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:36.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:23:36.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:36.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:36.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:36.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:36.335 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:23:36.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:36.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:36.863 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 46 up_thru 51 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/191617378,v1:127.0.0.1:6803/191617378] [v2:127.0.0.1:6804/191617378,v1:127.0.0.1:6805/191617378] exists,up e7399d45-bbbb-4451-b676-87112ca8a79b 2024-04-25T07:23:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-04-25T07:23:36.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:23:36.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:36.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:37.418 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/2149456914,v1:127.0.0.1:6819/2149456914] [v2:127.0.0.1:6820/2149456914,v1:127.0.0.1:6821/2149456914] exists,up 39207523-3cd8-471c-b010-5a75fbb25cb6 2024-04-25T07:23:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-04-25T07:23:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:23:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:37.420 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:23:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:37.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:37.904 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/3001253986,v1:127.0.0.1:6811/3001253986] [v2:127.0.0.1:6812/3001253986,v1:127.0.0.1:6813/3001253986] exists,up 8700d637-8430-4fba-817a-dba36e87e380 2024-04-25T07:23:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:38.318 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:38.318 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:23:37.233611+0000 43'102 59:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:23:38.318 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:38.318 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:38.318 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:38.333 INFO:tasks.workunit.client.0.smithi031.stdout:wait for peering 2024-04-25T07:23:38.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-04-25T07:23:38.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:38.775 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:38.775 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:23:37.233611+0000 43'102 59:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:23:38.775 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:38.775 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:38.776 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:38.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-04-25T07:23:38.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-04-25T07:23:38.923 INFO:tasks.workunit.client.0.smithi031.stdout:killing divergent 1 2024-04-25T07:23:38.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:39.342 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:39.342 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:23:37.233611+0000 43'102 59:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:23:39.342 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:39.342 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:39.342 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:39.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:23:39.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:23:39.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:23:39.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:23:39.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:23:39.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:23:39.468 INFO:tasks.workunit.client.0.smithi031.stdout:reviving divergent 1 2024-04-25T07:23:39.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:23:39.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-04-25T07:23:39.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:39.889 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:39.889 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:23:37.233611+0000 43'102 59:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:23:39.889 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:23:39.889 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:39.890 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:23:39.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:23:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:23:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:23:39.914 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:23:39.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:23:39.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=edeed592-e028-4157-aeea-12aafe002180 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.25959/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:23:39.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:23:39.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:23:39.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:23:39.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:23:39.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:23:39.952 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:39.954+0000 7f24566d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:39.954 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:39.954+0000 7f24566d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:39.955 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:39.958+0000 7f24566d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:40.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:23:40.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:40.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:23:40.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:40.349 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:23:40.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:40.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:40.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:40.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:40.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:40.514+0000 7f24566d57c0 -1 Falling back to public interface 2024-04-25T07:23:40.817 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/2149456914,v1:127.0.0.1:6819/2149456914] [v2:127.0.0.1:6820/2149456914,v1:127.0.0.1:6821/2149456914] exists,up 39207523-3cd8-471c-b010-5a75fbb25cb6 2024-04-25T07:23:40.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:40.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:40.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:40.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-04-25T07:23:43.355 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:23:43.358+0000 7f24566d57c0 -1 osd.1 59 log_to_monitors true 2024-04-25T07:24:00.820 INFO:tasks.workunit.client.0.smithi031.stdout:allowing recovery 2024-04-25T07:24:00.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-04-25T07:24:00.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:01.249 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:01.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:23:45.551746+0000 59'103 64:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:34:44.826851+0000 0 0 2024-04-25T07:24:01.249 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:24:01.249 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:01.249 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:24:01.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:01.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:24:01.426 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:24:01.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:01.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:24:01.598 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:24:01.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:01.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:24:01.767 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:24:01.782 INFO:tasks.workunit.client.0.smithi031.stdout:reading divergent objects 2024-04-25T07:24:01.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-04-25T07:24:01.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:02.191 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:02.191 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:23:45.551746+0000 59'103 64:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:20:50.490235+0000 0'0 2024-04-25T07:20:50.490235+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:34:44.826851+0000 0 0 2024-04-25T07:24:02.191 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T07:24:02.191 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:02.191 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T07:24:02.209 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:24:02.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-04-25T07:24:02.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:24:02.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:24:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:24:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:24:02.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:24:02.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:24:02.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:24:02.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:24:02.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-04-25T07:24:02.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:24:02.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:24:02.377 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:21:28.637987+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:24:02.377 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:21:28.637987+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4645.0:1 2024-04-25T07:21:35.464883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4645.0:1 2024-04-25T07:21:35.464883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-04-25T07:24:02.378 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4645.0:1 2024-04-25T07:21:35.464883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4636.0:1 2024-04-25T07:21:34.245852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4642.0:1 2024-04-25T07:21:35.249545+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4636.0:1 2024-04-25T07:21:34.245852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4642.0:1 2024-04-25T07:21:35.249545+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-04-25T07:24:02.379 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4642.0:1 2024-04-25T07:21:35.249545+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:24:02.380 INFO:tasks.workunit.client.0.smithi031.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:23:37.126+0000 7fc047ef3640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:24:02.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:24:02.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:24:02.386 INFO:tasks.workunit.client.0.smithi031.stdout:success 2024-04-25T07:24:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-04-25T07:24:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-04-25T07:24:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:24:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:24:02.890 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T07:24:02.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-04-25T07:24:02.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:02.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:02.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:02.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:02.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:03.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:03.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:03.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:24:03.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:24:03.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:24:03.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:24:03.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:24:03.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:24:03.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:03.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:24:03.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:24:03.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:03.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:24:03.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:24:03.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:24:03.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:24:03.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:24:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:24:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:24:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:24:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:24:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/divergent-priors 0 2024-04-25T07:24:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:03.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:03.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:03.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:24:03.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:24:03.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:24:03.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:24:03.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:24:03.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:24:03.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:03.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:24:03.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:24:03.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:24:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:24:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:24:03.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:24:03.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25959 2024-04-25T07:24:03.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25959 2024-04-25T07:24:03.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:24:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:24:03.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:24:03.066 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:24:03.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:24:03.114 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-04-25T07:24:03.115 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-04-25T07:24:03.167 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:24:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:24:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:24:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:24:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:24:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:24:03.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ec-error-rollforward 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-04-25T07:24:03.172 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-04-25T07:24:03.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none ' 2024-04-25T07:24:03.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-04-25T07:24:03.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-04-25T07:24:03.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-04-25T07:24:03.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-04-25T07:24:03.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-04-25T07:24:03.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-04-25T07:24:03.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-04-25T07:24:03.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-04-25T07:24:03.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:03.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:03.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:03.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:24:03.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:24:03.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:24:03.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:24:03.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:24:03.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:24:03.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:03.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:24:03.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:24:03.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:03.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:24:03.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:24:03.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:24:03.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:24:03.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.82552 2024-04-25T07:24:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:24:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:24:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-04-25T07:24:03.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:24:03.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:03.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.82552 2024-04-25T07:24:03.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-04-25T07:24:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-04-25T07:24:03.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:24:03.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:03.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:03.640 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:03.640 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.640 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:03.640 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:24:03.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:24:03.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:24:03.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:24:03.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:24:03.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:24:03.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:24:03.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:24:03.870 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.82552/ceph-mon.a.asok 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:24:03.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.82552/ceph-mon.a.asok config get fsid 2024-04-25T07:24:03.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:24:03.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:24:03.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:24:03.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:24:03.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:24:03.987 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:24:03.987 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:24:03.987 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:24:03.988 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:24:03.988 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:03.988 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:03.989 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.82552/ceph-mon.a.asok 2024-04-25T07:24:03.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:24:03.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.82552/ceph-mon.a.asok config get mon_host 2024-04-25T07:24:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-04-25T07:24:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-04-25T07:24:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:24:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:24:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:24:04.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-04-25T07:24:04.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:24:04.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:24:04.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:04.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:04.998 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:04.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:24:04.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:24:04.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:24:05.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-04-25T07:24:05.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:24:05.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:24:05.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:24:05.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03b45c9d-3412-494d-ba1b-a3ad75e8988f 2024-04-25T07:24:05.032 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 03b45c9d-3412-494d-ba1b-a3ad75e8988f 2024-04-25T07:24:05.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 03b45c9d-3412-494d-ba1b-a3ad75e8988f' 2024-04-25T07:24:05.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:24:05.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVBSpmLucIAxAA8R5s0BIYbDM0oWzSIUHTBw== 2024-04-25T07:24:05.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVBSpmLucIAxAA8R5s0BIYbDM0oWzSIUHTBw=="}' 2024-04-25T07:24:05.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03b45c9d-3412-494d-ba1b-a3ad75e8988f -i td/ec-error-rollforward/0/new.json 2024-04-25T07:24:05.347 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:24:05.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-04-25T07:24:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVBSpmLucIAxAA8R5s0BIYbDM0oWzSIUHTBw== --osd-uuid 03b45c9d-3412-494d-ba1b-a3ad75e8988f 2024-04-25T07:24:05.390 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.389+0000 7f09dc1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:05.413 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.413+0000 7f09dc1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:05.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.417+0000 7f09dc1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:05.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.417+0000 7f09dc1cd7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:24:05.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.417+0000 7f09dc1cd7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:24:05.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.417+0000 7f09dc1cd7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:24:05.445 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:05.445+0000 7f09dc1cd7c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-04-25T07:24:11.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-04-25T07:24:11.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:24:11.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:24:11.936 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:24:11.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:24:12.509 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:24:12.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:24:12.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:24:12.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:24:12.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:24:12.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:24:12.540 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:12.541+0000 7fdd30db47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:12.585 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:12.585+0000 7fdd30db47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:12.586 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:12.589+0000 7fdd30db47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:13.024 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:24:13.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:24:13.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:13.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:13.156 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:13.157+0000 7fdd30db47c0 -1 Falling back to public interface 2024-04-25T07:24:13.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:14.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:14.509 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:24:14.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:14.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:24:14.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:14.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:15.797 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:15.797+0000 7fdd30db47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:24:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:15.933 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:24:15.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:24:15.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:15.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:16.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:17.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:17.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:17.706 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:24:17.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:24:17.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:17.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:18.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:19.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:19.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:19.192 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:24:19.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:24:19.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:19.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:19.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:20.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:20.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:20.630 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:24:20.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:24:20.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:20.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:21.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:22.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:22.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:22.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:24:22.122 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:24:22.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:22.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:23.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:23.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:23.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:24:23.553 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:24:23.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:23.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:24.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:25.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:25.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:25.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:24:25.005 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:24:25.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:25.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:25.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:24:26.444 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:24:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:26.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:27.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:27.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:27.997 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:24:27.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:24:27.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:27.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:28.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:29.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:29.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:29.439 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:24:29.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:24:29.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:29.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:30.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:31.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:31.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:31.794 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:24:31.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:24:31.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:31.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:32.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:33.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:33.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:33.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:24:33.466 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:24:33.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:33.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:34.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:35.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:35.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:35.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:24:35.379 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:24:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:36.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:37.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:37.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:37.021 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:24:37.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:24:37.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:37.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:37.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:38.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:38.695 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:24:38.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:24:38.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:38.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:39.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:40.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:40.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:40.473 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:24:40.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:24:40.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:40.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:40.897 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3883457373,v1:127.0.0.1:6803/3883457373] [v2:127.0.0.1:6804/3883457373,v1:127.0.0.1:6805/3883457373] exists,up 03b45c9d-3412-494d-ba1b-a3ad75e8988f 2024-04-25T07:24:40.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:24:40.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:24:40.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:24:40.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-04-25T07:24:40.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:24:40.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:24:40.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:24:40.903 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:24:40.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e5d2036-0843-43ad-a7ef-757bf44d9d8c 2024-04-25T07:24:40.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7e5d2036-0843-43ad-a7ef-757bf44d9d8c' 2024-04-25T07:24:40.904 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 7e5d2036-0843-43ad-a7ef-757bf44d9d8c 2024-04-25T07:24:40.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:24:40.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4BSpmt9z4NhAA/ikaREi5ey6fCdHQmAOrSA== 2024-04-25T07:24:40.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4BSpmt9z4NhAA/ikaREi5ey6fCdHQmAOrSA=="}' 2024-04-25T07:24:40.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e5d2036-0843-43ad-a7ef-757bf44d9d8c -i td/ec-error-rollforward/1/new.json 2024-04-25T07:24:41.468 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:24:41.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-04-25T07:24:41.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4BSpmt9z4NhAA/ikaREi5ey6fCdHQmAOrSA== --osd-uuid 7e5d2036-0843-43ad-a7ef-757bf44d9d8c 2024-04-25T07:24:41.522 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.520+0000 7f71422cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:41.537 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.536+0000 7f71422cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:41.539 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.540+0000 7f71422cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:41.539 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.540+0000 7f71422cc7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:24:41.540 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.540+0000 7f71422cc7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:24:41.540 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.540+0000 7f71422cc7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:24:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:41.576+0000 7f71422cc7c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-04-25T07:24:48.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-04-25T07:24:48.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:24:48.402 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:24:48.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:24:48.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:24:48.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:24:48.985 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:24:48.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:24:48.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:24:48.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:24:48.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:24:49.014 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:49.012+0000 7f5ab70d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:49.017 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:49.016+0000 7f5ab70d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:49.018 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:49.016+0000 7f5ab70d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:49.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:24:49.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:24:49.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:49.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:49.580 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:49.580+0000 7f5ab70d27c0 -1 Falling back to public interface 2024-04-25T07:24:49.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:50.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:50.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:50.926 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:24:50.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:24:50.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:50.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:51.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:52.257 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:24:52.256+0000 7f5ab70d27c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:24:52.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:52.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:52.361 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:24:52.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:24:52.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:52.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:53.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:54.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:54.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:54.022 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:24:54.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:24:54.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:54.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:54.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:55.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:55.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:55.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:24:55.564 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:24:55.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:55.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:56.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:57.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:57.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:57.176 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:24:57.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:24:57.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:57.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:57.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:58.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:58.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:58.605 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:24:58.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:24:58.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:58.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:24:59.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:00.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:00.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:00.150 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:25:00.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:25:00.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:00.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:00.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:02.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:03.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:03.216 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:25:03.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:03.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:25:03.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:03.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:03.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:04.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:04.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:04.711 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:25:04.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:25:04.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:04.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:05.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:06.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:06.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:06.219 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:25:06.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:25:06.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:06.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:06.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:07.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:07.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:07.897 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:25:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:25:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:08.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:09.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:09.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:09.507 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:25:09.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:25:09.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:09.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:10.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:11.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:11.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:11.264 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:25:11.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:25:11.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:11.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:12.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:13.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:13.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:13.041 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:25:13.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:25:13.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:13.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:13.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:14.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:14.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:14.823 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:25:14.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:25:14.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:14.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:15.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:16.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:16.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:16.404 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:25:16.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:25:16.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:16.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:25:16.940 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/852605838,v1:127.0.0.1:6811/852605838] [v2:127.0.0.1:6812/852605838,v1:127.0.0.1:6813/852605838] exists,up 7e5d2036-0843-43ad-a7ef-757bf44d9d8c 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:25:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:25:16.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:16.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:16.945 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:16.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d60dbbdc-516b-4318-90f8-4d7da35ba695 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 d60dbbdc-516b-4318-90f8-4d7da35ba695 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d60dbbdc-516b-4318-90f8-4d7da35ba695' 2024-04-25T07:25:16.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:25:16.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcBSpmMEeeORAAqk18w5izQVcwk+bhWQUspw== 2024-04-25T07:25:16.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcBSpmMEeeORAAqk18w5izQVcwk+bhWQUspw=="}' 2024-04-25T07:25:16.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d60dbbdc-516b-4318-90f8-4d7da35ba695 -i td/ec-error-rollforward/2/new.json 2024-04-25T07:25:17.494 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:25:17.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-04-25T07:25:17.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcBSpmMEeeORAAqk18w5izQVcwk+bhWQUspw== --osd-uuid d60dbbdc-516b-4318-90f8-4d7da35ba695 2024-04-25T07:25:17.553 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.555+0000 7f12b4d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:17.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.739+0000 7f12b4d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:17.740 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.743+0000 7f12b4d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:17.740 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.743+0000 7f12b4d0e7c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:25:17.740 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.743+0000 7f12b4d0e7c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:25:17.740 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.743+0000 7f12b4d0e7c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:25:17.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:17.783+0000 7f12b4d0e7c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-04-25T07:25:24.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-04-25T07:25:24.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:25:24.516 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:25:24.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:25:24.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:25:25.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:25:25.085 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:25:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:25:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:25:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:25:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:25:25.114 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:25.115+0000 7f052c8a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:25.149 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:25.151+0000 7f052c8a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:25.150 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:25.151+0000 7f052c8a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:25.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:25:25.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:25:25.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:25:25.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:25:25.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:25:25.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:25.538 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:25:25.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:25:25.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:25.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:25.722 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:25.723+0000 7f052c8a47c0 -1 Falling back to public interface 2024-04-25T07:25:25.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:26.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:26.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:26.969 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:25:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:25:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:26.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:27.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:28.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:28.391 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:25:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:25:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:28.535 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:28.539+0000 7f052c8a47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:25:28.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:29.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:29.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:29.893 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:25:29.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:25:29.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:29.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:30.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:31.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:31.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:31.418 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:25:31.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:25:31.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:31.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:31.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:32.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:32.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:32.860 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:25:32.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:25:32.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:32.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:33.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:34.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:34.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:34.397 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:25:34.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:25:34.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:34.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:34.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:35.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:35.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:35.827 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:25:35.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:25:35.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:35.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:36.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:37.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:37.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:37.424 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:25:37.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:25:37.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:37.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:37.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:38.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:38.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:38.885 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:25:38.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:25:38.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:38.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:39.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:40.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:40.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:40.437 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:25:40.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:25:40.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:40.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:40.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:41.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:41.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:41.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:25:41.908 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:25:41.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:41.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:42.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:43.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:43.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:43.356 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:25:43.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:25:43.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:43.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:43.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:44.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:44.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:44.867 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:25:44.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:25:44.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:44.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:45.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:46.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:46.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:46.636 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:25:46.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:25:46.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:46.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:47.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:48.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:48.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:48.673 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:25:48.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:25:48.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:48.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:49.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:50.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:50.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:50.562 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:25:50.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:25:50.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:50.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:51.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:52.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:52.460 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:25:52.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:52.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:25:52.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:52.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:52.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:53.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:53.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:54.000 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:25:54.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:25:54.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:54.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:54.440 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1573044002,v1:127.0.0.1:6819/1573044002] [v2:127.0.0.1:6820/1573044002,v1:127.0.0.1:6821/1573044002] exists,up d60dbbdc-516b-4318-90f8-4d7da35ba695 2024-04-25T07:25:54.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:54.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:54.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:25:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:25:54.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d03da3d-4b9d-49c0-9cc8-533dc5ea3166 2024-04-25T07:25:54.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3d03da3d-4b9d-49c0-9cc8-533dc5ea3166' 2024-04-25T07:25:54.447 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 3d03da3d-4b9d-49c0-9cc8-533dc5ea3166 2024-04-25T07:25:54.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:25:54.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCBSpmT1oLHBAA2u6jFY2INWnybbZjb7jnpg== 2024-04-25T07:25:54.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCBSpmT1oLHBAA2u6jFY2INWnybbZjb7jnpg=="}' 2024-04-25T07:25:54.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d03da3d-4b9d-49c0-9cc8-533dc5ea3166 -i td/ec-error-rollforward/3/new.json 2024-04-25T07:25:54.979 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:25:54.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-04-25T07:25:54.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCBSpmT1oLHBAA2u6jFY2INWnybbZjb7jnpg== --osd-uuid 3d03da3d-4b9d-49c0-9cc8-533dc5ea3166 2024-04-25T07:25:55.030 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.031+0000 7fb7b1d917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:55.090 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.091+0000 7fb7b1d917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:55.092 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.095+0000 7fb7b1d917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:55.092 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.095+0000 7fb7b1d917c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:25:55.093 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.095+0000 7fb7b1d917c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:25:55.093 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.095+0000 7fb7b1d917c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:25:55.138 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:25:55.139+0000 7fb7b1d917c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-04-25T07:26:01.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-04-25T07:26:01.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:26:01.853 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T07:26:01.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:26:01.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:26:02.435 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T07:26:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:26:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:26:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:26:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:26:02.466 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:02.466+0000 7f91f08417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:02.470 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:02.470+0000 7f91f08417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:02.472 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:02.474+0000 7f91f08417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:02.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:02.903 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:26:02.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:02.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:03.045 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:03.046+0000 7f91f08417c0 -1 Falling back to public interface 2024-04-25T07:26:03.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:04.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:04.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:04.330 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:26:04.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:26:04.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:04.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:04.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:05.595 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:05.594+0000 7f91f08417c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:26:05.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:05.755 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:26:05.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:05.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:26:05.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:05.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:06.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:07.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:07.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:07.343 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:26:07.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:26:07.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:07.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:07.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:08.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:08.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:08.790 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:26:08.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:26:08.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:08.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:09.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:10.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:10.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:10.255 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:26:10.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:26:10.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:10.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:10.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:11.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:11.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:11.748 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:26:11.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:26:11.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:11.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:12.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:13.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:13.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:13.378 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:26:13.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:26:13.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:13.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:13.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:14.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:14.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:14.830 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:26:14.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:26:14.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:14.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:15.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:16.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:16.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:16.295 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:26:16.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:26:16.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:16.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:16.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:17.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:17.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:17.778 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:26:17.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:26:17.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:17.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:18.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:19.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:19.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:19.240 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:26:19.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:26:19.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:19.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:20.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:21.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:21.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:21.283 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:26:21.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:26:21.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:21.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:22.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:23.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:23.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:23.108 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:26:23.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:26:23.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:23.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:23.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:24.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:24.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:24.913 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:26:24.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:26:24.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:24.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:25.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:26.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:26.919 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:26:26.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:26.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:26:26.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:26.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:27.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:28.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:28.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:28.725 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:26:28.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:26:28.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:28.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:29.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:30.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:30.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:30.692 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:26:30.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:26:30.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:30.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:31.187 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1884111223,v1:127.0.0.1:6827/1884111223] [v2:127.0.0.1:6828/1884111223,v1:127.0.0.1:6829/1884111223] exists,up 3d03da3d-4b9d-49c0-9cc8-533dc5ea3166 2024-04-25T07:26:31.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:31.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:31.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:31.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-04-25T07:26:31.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-04-25T07:26:32.965 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'ec' already exists 2024-04-25T07:26:32.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-04-25T07:26:34.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-04-25T07:26:34.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 87112 2024-04-25T07:26:34.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=89848 2024-04-25T07:26:34.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:26:34.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-04-25T07:26:35.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 89869' 2024-04-25T07:26:36.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-04-25T07:26:36.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 89870' 2024-04-25T07:26:36.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-04-25T07:26:36.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 89871' 2024-04-25T07:26:36.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:26:36.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-04-25T07:26:36.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-04-25T07:26:36.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:36.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:36.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:36.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:36.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:37.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:37.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 89848 89869 89870 89871 2024-04-25T07:26:37.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:37.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:37.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:26:37.040 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:26:37.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:26:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:37.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-04-25T07:26:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:26:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:37.064 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.066+0000 7f30f3ebe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:37.067 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.066+0000 7f30f3ebe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:37.068 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.070+0000 7f30f3ebe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:37.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:26:37.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:37.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:26:37.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:37.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:37.455 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:26:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.614+0000 7f30f3ebe7c0 -1 Falling back to public interface 2024-04-25T07:26:37.903 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3883457373,v1:127.0.0.1:6803/3883457373] [v2:127.0.0.1:6804/3883457373,v1:127.0.0.1:6805/3883457373] exists,up 03b45c9d-3412-494d-ba1b-a3ad75e8988f 2024-04-25T07:26:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-04-25T07:26:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:26:37.909 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:26:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:26:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:37.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-04-25T07:26:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:26:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:37.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:37.930 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.930+0000 7ffa917c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:37.957 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.958+0000 7ffa917c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:37.958 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:37.958+0000 7ffa917c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:38.335 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:26:38.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:38.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:38.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:26:38.505 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:38.506+0000 7ffa917c97c0 -1 Falling back to public interface 2024-04-25T07:26:38.769 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/852605838,v1:127.0.0.1:6811/852605838] [v2:127.0.0.1:6812/852605838,v1:127.0.0.1:6813/852605838] exists,up 7e5d2036-0843-43ad-a7ef-757bf44d9d8c 2024-04-25T07:26:38.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:38.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:26:38.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:26:38.775 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:26:38.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:26:38.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:38.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-04-25T07:26:38.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:26:38.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:38.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:38.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:38.802 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:38.802+0000 7fae454097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:38.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:38.810+0000 7fae454097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:38.810 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:38.810+0000 7fae454097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:39.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:26:39.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:39.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:39.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:26:39.381 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:39.382+0000 7fae454097c0 -1 Falling back to public interface 2024-04-25T07:26:39.623 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1573044002,v1:127.0.0.1:6819/1573044002] [v2:127.0.0.1:6820/1573044002,v1:127.0.0.1:6821/1573044002] exists,up d60dbbdc-516b-4318-90f8-4d7da35ba695 2024-04-25T07:26:39.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:39.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:26:39.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:39.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:26:39.628 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T07:26:39.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:26:39.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=5bb722db-6cab-454f-98b1-2cc8b1c69710 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:39.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-04-25T07:26:39.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:26:39.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:39.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:39.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:39.657 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:39.658+0000 7fccf41d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:39.660 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:39.662+0000 7fccf41d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:39.662 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:39.662+0000 7fccf41d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:40.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:26:40.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:40.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:26:40.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:40.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:40.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:40.096 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:26:40.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:40.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:40.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:26:40.213 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:40.214+0000 7fccf41d97c0 -1 Falling back to public interface 2024-04-25T07:26:40.539 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1884111223,v1:127.0.0.1:6827/1884111223] [v2:127.0.0.1:6828/1884111223,v1:127.0.0.1:6829/1884111223] exists,up 3d03da3d-4b9d-49c0-9cc8-533dc5ea3166 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:26:40.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:26:40.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:26:40.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:26:40.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:26:40.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:26:40.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:26:40.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:26:40.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:26:40.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:26:40.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:26:40.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:26:40.622 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:41.035 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:26:41.377 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:41.378+0000 7f30f3ebe7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T07:26:42.750 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:42.749+0000 7ffa917c97c0 -1 osd.1 26 log_to_monitors true 2024-04-25T07:26:43.694 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:43.693+0000 7fae454097c0 -1 osd.2 25 log_to_monitors true 2024-04-25T07:26:44.250 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:44.249+0000 7fccf41d97c0 -1 osd.3 26 log_to_monitors true 2024-04-25T07:26:46.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051587 2024-04-25T07:26:46.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051587 2024-04-25T07:26:46.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587' 2024-04-25T07:26:46.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:46.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:26:46.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953475 2024-04-25T07:26:46.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953475 2024-04-25T07:26:46.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-04-25T07:26:46.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:46.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:26:46.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822659 2024-04-25T07:26:46.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822659 2024-04-25T07:26:46.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659' 2024-04-25T07:26:46.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:46.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:26:46.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T07:26:46.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T07:26:46.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659 3-163208757251' 2024-04-25T07:26:46.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:46.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051587 2024-04-25T07:26:46.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:46.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:26:46.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051587 2024-04-25T07:26:46.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:46.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051587 2024-04-25T07:26:46.943 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 124554051587 2024-04-25T07:26:46.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-04-25T07:26:46.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:26:47.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 124554051587 2024-04-25T07:26:47.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:26:48.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:26:48.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:26:48.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051588 -lt 124554051587 2024-04-25T07:26:48.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:48.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-137438953475 2024-04-25T07:26:48.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:48.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:26:48.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:48.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-137438953475 2024-04-25T07:26:48.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953475 2024-04-25T07:26:48.911 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 137438953475 2024-04-25T07:26:48.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-04-25T07:26:48.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:26:49.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953475 -lt 137438953475 2024-04-25T07:26:49.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:49.399 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-154618822659 2024-04-25T07:26:49.399 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:49.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:26:49.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-154618822659 2024-04-25T07:26:49.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822659 2024-04-25T07:26:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-04-25T07:26:49.403 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 154618822659 2024-04-25T07:26:49.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:26:49.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822659 -lt 154618822659 2024-04-25T07:26:49.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:49.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-163208757251 2024-04-25T07:26:49.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:49.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:26:49.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:49.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-163208757251 2024-04-25T07:26:49.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T07:26:49.855 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 163208757251 2024-04-25T07:26:49.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 163208757251' 2024-04-25T07:26:49.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:26:50.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757251 -lt 163208757251 2024-04-25T07:26:50.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:26:50.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:50.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:50.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:26:50.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:50.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:50.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:50.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:50.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:50.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:50.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:51.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:26:51.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:51.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:51.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:26:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:26:51.830 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:51.830 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:51.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:51.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:51.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:52.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:52.142 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:52.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:52.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:52.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:52.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:52.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:26:52.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:26:52.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:52.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:52.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:26:52.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.82552 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ec-error-rollforward 0 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:26:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:26:52.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:52.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:52.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:52.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:52.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:52.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:52.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:52.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:52.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:52.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:52.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:52.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:52.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:52.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:26:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:26:52.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:52.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:52.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82552 2024-04-25T07:26:52.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.82552 2024-04-25T07:26:52.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:52.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:52.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:26:52.191 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:26:52.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:26:52.243 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-04-25T07:26:52.243 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-04-25T07:26:52.301 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:26:52.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:26:52.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:26:52.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:26:52.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:26:52.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:26:52.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bench 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-04-25T07:26:52.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-04-25T07:26:52.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=a53efb9e-f453-4a44-bdf4-c075e9ad106b --auth-supported=none ' 2024-04-25T07:26:52.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-04-25T07:26:52.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-04-25T07:26:52.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-04-25T07:26:52.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:26:52.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-04-25T07:26:52.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-04-25T07:26:52.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:52.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:52.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:52.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:52.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:52.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:52.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:52.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:52.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:52.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:52.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:52.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:52.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:52.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:52.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:26:52.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:26:52.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:52.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:52.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:52.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92375 2024-04-25T07:26:52.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:52.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:52.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-04-25T07:26:52.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:26:52.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:52.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:52.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.92375 2024-04-25T07:26:52.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-04-25T07:26:52.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-04-25T07:26:52.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:26:52.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:52.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:52.810 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:52.810 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:52.810 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:52.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92375/$cluster-$name.asok' 2024-04-25T07:26:52.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92375/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:26:53.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:26:53.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:26:53.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:26:53.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:26:53.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:26:53.022 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:26:53.022 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:26:53.022 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92375/ceph-mon.a.asok 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:26:53.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.92375/ceph-mon.a.asok config get fsid 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:26:53.138 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:26:53.139 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:53.139 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:53.139 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:53.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92375/ceph-mon.a.asok 2024-04-25T07:26:53.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:26:53.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.92375/ceph-mon.a.asok config get mon_host 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-04-25T07:26:53.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:26:54.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:26:54.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:54.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:54.152 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:54.152 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:54.152 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:54.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92375/$cluster-$name.asok' 2024-04-25T07:26:54.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:26:54.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92375/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a53efb9e-f453-4a44-bdf4-c075e9ad106b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92375/$cluster-$name.asok' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92375/$cluster-$name.asok' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a9cdcca-5644-414b-9415-63dc33a40276 2024-04-25T07:26:54.190 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 5a9cdcca-5644-414b-9415-63dc33a40276 2024-04-25T07:26:54.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5a9cdcca-5644-414b-9415-63dc33a40276' 2024-04-25T07:26:54.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:26:54.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+BSpmqz4mDBAAJW28O6kY+zPUjwcxqWYolw== 2024-04-25T07:26:54.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+BSpmqz4mDBAAJW28O6kY+zPUjwcxqWYolw=="}' 2024-04-25T07:26:54.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a9cdcca-5644-414b-9415-63dc33a40276 -i td/osd-bench/0/new.json 2024-04-25T07:26:54.506 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:26:54.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-04-25T07:26:54.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a53efb9e-f453-4a44-bdf4-c075e9ad106b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92375/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+BSpmqz4mDBAAJW28O6kY+zPUjwcxqWYolw== --osd-uuid 5a9cdcca-5644-414b-9415-63dc33a40276 2024-04-25T07:26:54.550 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.549+0000 7f2bc4b5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:54.583 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.581+0000 7f2bc4b5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:54.584 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.585+0000 7f2bc4b5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:54.585 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.585+0000 7f2bc4b5a7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:26:54.585 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.585+0000 7f2bc4b5a7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:26:54.585 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.585+0000 7f2bc4b5a7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:26:54.623 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:26:54.621+0000 7f2bc4b5a7c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-04-25T07:27:01.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-04-25T07:27:01.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:27:01.177 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:27:01.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:27:01.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:27:01.752 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:27:01.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:27:01.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a53efb9e-f453-4a44-bdf4-c075e9ad106b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92375/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:01.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:27:01.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:27:01.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:27:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:27:01.785+0000 7fb81efa17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:01.805 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:27:01.805+0000 7fb81efa17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:01.807 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:27:01.805+0000 7fb81efa17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:02.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:27:02.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:02.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:27:02.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:02.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:02.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:02.187 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:27:02.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:02.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:02.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:02.385 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:27:02.385+0000 7fb81efa17c0 -1 Falling back to public interface 2024-04-25T07:27:02.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:03.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:03.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:03.610 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:27:03.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:27:03.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:03.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:04.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:04.757 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:27:04.757+0000 7fb81efa17c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:27:05.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:05.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:05.036 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:27:05.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:27:05.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:05.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:05.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:06.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:06.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:06.512 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:27:06.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:27:06.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:06.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:06.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:07.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:07.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:07.979 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:27:07.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:27:07.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:07.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:08.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:09.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:09.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:09.527 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:27:09.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:27:09.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:09.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:10.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:10.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:10.994 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:27:10.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:27:10.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:10.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:11.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:12.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:12.604 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:27:12.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:12.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:27:12.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:12.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:13.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:14.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:14.038 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:27:14.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:14.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:27:14.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:14.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:14.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:15.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:15.556 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:27:15.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:15.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:27:15.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:15.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:16.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:17.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:17.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:17.020 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:27:17.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:27:17.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:17.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:17.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:18.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:18.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:18.840 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:27:18.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:27:18.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:18.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:20.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:21.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:21.060 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:27:21.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:21.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:27:21.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:21.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:21.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:22.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:22.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:22.993 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:27:22.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:27:22.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:22.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:23.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:24.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:24.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:24.737 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:27:24.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:27:24.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:24.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:25.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:26.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:26.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:26.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:27:26.251 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:27:26.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:26.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:27.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:28.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:28.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:28.261 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:27:28.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:27:28.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:28.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:29.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:30.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:30.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:30.179 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:27:30.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:27:30.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:30.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:30.607 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1867135336,v1:127.0.0.1:6803/1867135336] [v2:127.0.0.1:6804/1867135336,v1:127.0.0.1:6805/1867135336] exists,up 5a9cdcca-5644-414b-9415-63dc33a40276 2024-04-25T07:27:30.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:30.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:30.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:30.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-04-25T07:27:30.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-04-25T07:27:30.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-04-25T07:27:30.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-04-25T07:27:30.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-04-25T07:27:30.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-04-25T07:27:30.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-04-25T07:27:30.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-04-25T07:27:30.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-04-25T07:27:30.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-04-25T07:27:30.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_max_block_size 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:27:30.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-04-25T07:27:30.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:27:30.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:27:30.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-04-25T07:27:30.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:27:30.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-04-25T07:27:30.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-04-25T07:27:30.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-04-25T07:27:30.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:27:30.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:27:30.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=bench_small_size_max_iops 2024-04-25T07:27:30.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:27:30.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:27:30.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-04-25T07:27:30.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:27:30.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:27:30.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:27:30.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:27:31.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:27:31.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:27:31.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-04-25T07:27:31.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:27:31.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-04-25T07:28:25.580 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:28:25.580 INFO:tasks.workunit.client.0.smithi031.stdout: "bytes_written": 1073741824, 2024-04-25T07:28:25.580 INFO:tasks.workunit.client.0.smithi031.stdout: "blocksize": 4194304, 2024-04-25T07:28:25.581 INFO:tasks.workunit.client.0.smithi031.stdout: "elapsed_sec": 25.137795276999999, 2024-04-25T07:28:25.581 INFO:tasks.workunit.client.0.smithi031.stdout: "bytes_per_sec": 42714240.137933955, 2024-04-25T07:28:25.581 INFO:tasks.workunit.client.0.smithi031.stdout: "iops": 10.183868441089142 2024-04-25T07:28:25.581 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:28:25.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout: "bytes_written": 10, 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout: "blocksize": 14456, 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout: "elapsed_sec": 0.075481313999999994, 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout: "bytes_per_sec": 132.48312025940621, 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout: "iops": 0.0091645766643197422 2024-04-25T07:28:26.122 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:28:26.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-04-25T07:28:26.464 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T07:28:26.464 INFO:tasks.workunit.client.0.smithi031.stdout: "bytes_written": 1, 2024-04-25T07:28:26.464 INFO:tasks.workunit.client.0.smithi031.stdout: "blocksize": 14456, 2024-04-25T07:28:26.464 INFO:tasks.workunit.client.0.smithi031.stdout: "elapsed_sec": 0.041551880999999999, 2024-04-25T07:28:26.465 INFO:tasks.workunit.client.0.smithi031.stdout: "bytes_per_sec": 24.066299188717835, 2024-04-25T07:28:26.465 INFO:tasks.workunit.client.0.smithi031.stdout: "iops": 0.0016647965681182786 2024-04-25T07:28:26.465 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T07:28:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-04-25T07:28:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:28:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:28:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:28:26.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:28:26.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:28:26.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:28:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:28:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:28:26.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:28:26.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:28:26.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:28:26.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:28:26.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:28:26.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:28:26.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:28:26.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:28:26.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:28:26.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:28:26.811 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:28:26.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:28:26.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:28:26.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:28:26.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:28:26.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:26.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:28:26.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92375 2024-04-25T07:28:26.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:28:26.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:28:26.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:28:26.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bench 0 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:28:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:28:26.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:28:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:28:26.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:28:26.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:28:26.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:28:26.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:28:26.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:28:26.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:28:26.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:28:26.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:28:26.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:28:26.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:28:26.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:28:26.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:28:26.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:28:26.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:28:26.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:26.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92375 2024-04-25T07:28:26.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92375 2024-04-25T07:28:26.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:28:26.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:28:26.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:28:26.865 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:28:26.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:28:26.921 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-04-25T07:28:26.921 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-04-25T07:28:26.977 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:28:26.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bluefs-volume-ops 2024-04-25T07:28:26.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:26.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-04-25T07:28:26.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-04-25T07:28:26.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_bluestore2 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_bluestore_expand' 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:28:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:28:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:28:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:28:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:28:26.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:28:26.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:28:26.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:28:26.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:28:26.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:28:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:28:26.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:28:26.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:28:26.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:28:26.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:28:26.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:28:26.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:28:26.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:28:26.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:28:27.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:27.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:27.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:28:27.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:28:27.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:28:27.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:28:27.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:28:27.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:27.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:27.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94549 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-04-25T07:28:27.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:27.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-04-25T07:28:27.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-04-25T07:28:27.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-04-25T07:28:27.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:28:27.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:27.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:28:27.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:28:27.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:28:27.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:28:27.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:27.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:28:27.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:27.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:27.493 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:27.493 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:27.493 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:27.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:28:27.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:28:27.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:28:27.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:28:27.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:28:27.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:28:27.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:27.714 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94549/ceph-mon.a.asok 2024-04-25T07:28:27.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:28:27.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94549/ceph-mon.a.asok config get fsid 2024-04-25T07:28:27.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:28:27.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:28:27.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:28:27.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94549/ceph-mon.a.asok 2024-04-25T07:28:27.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:28:27.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94549/ceph-mon.a.asok config get mon_host 2024-04-25T07:28:27.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:28:27.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:27.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:28:27.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:28:27.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:28:27.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:28:27.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:28:28.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:28:28.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:28.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:28.899 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:28.899 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:28.899 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:28.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:28:28.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:28:28.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:28:28.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:28:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:28:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:28:28.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:28:28.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:28:28.972 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:28:28.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 937497c4-5df9-44b2-93ae-7997930c288d' 2024-04-25T07:28:28.973 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:28:28.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcBipmY0cVOxAA8DEStPcZl7UFXNeQkUxbDA== 2024-04-25T07:28:28.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcBipmY0cVOxAA8DEStPcZl7UFXNeQkUxbDA=="}' 2024-04-25T07:28:28.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 937497c4-5df9-44b2-93ae-7997930c288d -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:28:29.323 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:28:29.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:28:29.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcBipmY0cVOxAA8DEStPcZl7UFXNeQkUxbDA== --osd-uuid 937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:28:29.375 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.375+0000 7f836f4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:29.399 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.399+0000 7f836f4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:29.401 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.403+0000 7f836f4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:29.402 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.403+0000 7f836f4947c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:28:29.402 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.403+0000 7f836f4947c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:28:29.402 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.403+0000 7f836f4947c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:28:29.439 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:29.439+0000 7f836f4947c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:28:39.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:28:39.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:39.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:28:39.692 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:28:39.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:40.257 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:28:40.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:28:40.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:40.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:40.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:40.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:40.288 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:40.291+0000 7f951f2e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:40.291 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:40.291+0000 7f951f2e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:40.292 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:40.295+0000 7f951f2e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:40.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:28:40.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:40.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:28:40.739 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:28:40.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:40.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:40.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:40.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:40.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:40.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:41.327 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:41.327+0000 7f951f2e77c0 -1 Falling back to public interface 2024-04-25T07:28:42.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:42.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:42.203 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:28:42.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:28:42.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:42.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:43.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:43.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:43.632 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:28:43.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:28:43.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:43.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:44.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:45.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:45.074 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:28:45.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:45.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:28:45.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:45.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:45.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:46.237 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:28:46.239+0000 7f951f2e77c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:28:46.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:46.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:46.502 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:28:46.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:28:46.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:46.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:47.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:48.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:48.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:48.006 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:28:48.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:28:48.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:48.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:48.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:49.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:49.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:49.439 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:28:49.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:28:49.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:49.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:50.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:51.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:51.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:51.046 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:28:51.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:28:51.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:51.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:51.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:52.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:52.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:52.507 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:28:52.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:28:52.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:52.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:53.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:54.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:54.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:54.049 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:28:54.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:28:54.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:54.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:54.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:55.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:55.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:55.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:28:55.640 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:28:55.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:55.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:56.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:57.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:57.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:57.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:28:57.223 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:28:57.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:57.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:57.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:58.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:58.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:58.711 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:28:58.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:28:58.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:58.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:59.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:00.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:00.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:00.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:00.296 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:29:00.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:00.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:01.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:02.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:02.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:02.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:02.158 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:29:02.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:02.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:02.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:03.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:03.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:03.797 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:29:03.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:29:03.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:03.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:04.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:05.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:05.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:05.274 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:29:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:29:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:06.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:07.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:07.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:07.032 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:29:07.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:29:07.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:07.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:07.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:08.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:08.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:08.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:29:08.860 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:29:08.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:08.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:09.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:10.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:10.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:10.686 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:29:10.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:29:10.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:10.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:11.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:12.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:12.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:12.113 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T07:29:12.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:29:12.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:12.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:12.573 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/861091497,v1:127.0.0.1:6803/861091497] [v2:127.0.0.1:6804/861091497,v1:127.0.0.1:6805/861091497] exists,up 937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:29:12.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:12.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:12.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:12.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:29:12.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=96010 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:29:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:29:12.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:12.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:29:12.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:29:12.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:29:12.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:29:12.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:29:12.582 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:29:12.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cd0b508e-f02d-4fab-a50a-3024834d86ee' 2024-04-25T07:29:12.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:29:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIBipmVBwTJBAAvL4B9asCHqVlVCKncLkn3Q== 2024-04-25T07:29:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIBipmVBwTJBAAvL4B9asCHqVlVCKncLkn3Q=="}' 2024-04-25T07:29:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd0b508e-f02d-4fab-a50a-3024834d86ee -i td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:29:13.126 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:29:13.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:29:13.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIBipmVBwTJBAAvL4B9asCHqVlVCKncLkn3Q== --osd-uuid cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:29:13.170 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.170+0000 7fac429f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:13.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.174+0000 7fac429f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:13.177 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.178+0000 7fac429f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:13.178 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.178+0000 7fac429f07c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:29:13.178 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.178+0000 7fac429f07c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:29:13.178 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.178+0000 7fac429f07c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:29:13.216 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:13.214+0000 7fac429f07c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-04-25T07:29:23.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-04-25T07:29:23.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:29:23.996 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:29:23.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:29:23.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:29:24.580 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:29:24.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:29:24.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:24.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:29:24.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:29:24.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:29:24.612 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:24.610+0000 7ff86e3a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:24.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:24.654+0000 7ff86e3a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:24.657 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:24.658+0000 7ff86e3a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:25.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:29:25.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:25.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:29:25.077 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:29:25.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:25.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:25.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:25.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:25.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:25.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:25.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:25.707 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:25.706+0000 7ff86e3a97c0 -1 Falling back to public interface 2024-04-25T07:29:26.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:26.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:26.504 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:29:26.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:29:26.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:26.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:26.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:27.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:27.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:27.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:29:27.935 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:29:27.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:27.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:28.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:29.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:29.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:29.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:29:29.384 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:29:29.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:29.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:29.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:30.725 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:30.726+0000 7ff86e3a97c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:29:30.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:30.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:30.817 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:29:30.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:29:30.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:30.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:31.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:32.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:32.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:32.479 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:29:32.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:29:32.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:32.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:32.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:33.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:33.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:33.963 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:29:33.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:29:33.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:33.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:34.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:35.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:35.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:35.399 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:29:35.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:29:35.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:35.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:35.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:36.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:36.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:36.942 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:29:36.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:29:36.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:36.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:37.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:38.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:38.406 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:29:38.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:38.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:29:38.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:38.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:38.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:39.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:39.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:39.844 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:29:39.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:29:39.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:39.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:40.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:41.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:41.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:41.329 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:29:41.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:29:41.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:41.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:41.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:42.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:42.930 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:29:42.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:42.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:29:42.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:42.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:43.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:29:44.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:44.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:44.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:45.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:46.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:46.381 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:29:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:46.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:47.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:48.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:48.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:48.281 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:29:48.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:29:48.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:48.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:49.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:50.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:50.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:50.449 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:29:50.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:29:50.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:50.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:50.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:51.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:51.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:51.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:29:51.910 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:29:51.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:51.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:52.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:53.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:53.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:53.787 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:29:53.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:29:53.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:53.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:54.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:55.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:55.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:55.277 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:29:55.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:29:55.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:55.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:55.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:56.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:56.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:56.701 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T07:29:56.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:29:56.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:56.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:57.190 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1389348923,v1:127.0.0.1:6811/1389348923] [v2:127.0.0.1:6812/1389348923,v1:127.0.0.1:6813/1389348923] exists,up cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:29:57.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:57.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:57.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:57.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:29:57.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=98578 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:29:57.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:29:57.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:29:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:29:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:29:57.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:29:57.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:29:57.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c8e57ed6-531d-4749-a7ba-e3679de30c73' 2024-04-25T07:29:57.198 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:29:57.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:29:57.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1BipmM5tDDRAAT7jf2GDW7y6oneguZjKg4A== 2024-04-25T07:29:57.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1BipmM5tDDRAAT7jf2GDW7y6oneguZjKg4A=="}' 2024-04-25T07:29:57.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8e57ed6-531d-4749-a7ba-e3679de30c73 -i td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:29:57.753 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:29:57.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:29:57.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1BipmM5tDDRAAT7jf2GDW7y6oneguZjKg4A== --osd-uuid c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:29:57.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.801+0000 7f3e8ca6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:57.864 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.861+0000 7f3e8ca6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:57.866 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.865+0000 7f3e8ca6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:57.866 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.865+0000 7f3e8ca6d7c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:29:57.867 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.865+0000 7f3e8ca6d7c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:29:57.867 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.865+0000 7f3e8ca6d7c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:29:57.988 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:29:57.985+0000 7f3e8ca6d7c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-04-25T07:30:08.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-04-25T07:30:08.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:30:08.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:30:08.458 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:30:08.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:30:09.044 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:30:09.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:30:09.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:09.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:30:09.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:30:09.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:30:09.077 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:09.077+0000 7f111cc2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:09.104 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:09.101+0000 7f111cc2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:09.105 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:09.105+0000 7f111cc2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:09.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:30:09.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:09.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:30:09.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:09.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:09.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:09.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:09.485 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:30:09.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:09.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:09.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:10.167 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:10.165+0000 7f111cc2d7c0 -1 Falling back to public interface 2024-04-25T07:30:10.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:10.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:10.917 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:30:10.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:30:10.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:10.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:11.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:12.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:12.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:12.338 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:30:12.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:30:12.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:12.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:12.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:13.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:13.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:13.808 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:30:13.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:30:13.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:13.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:14.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:15.078 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:15.077+0000 7f111cc2d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:30:15.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:15.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:15.285 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:30:15.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:30:15.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:15.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:15.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:16.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:16.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:16.855 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:30:16.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:30:16.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:16.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:17.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:18.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:18.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:18.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:30:18.352 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:30:18.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:18.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:18.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:19.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:19.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:19.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:30:19.781 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:30:19.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:19.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:20.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:21.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:21.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:21.439 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:30:21.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:30:21.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:21.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:22.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:23.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:23.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:23.112 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:30:23.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:30:23.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:23.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:23.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:24.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:24.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:24.572 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:30:24.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:30:24.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:24.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:25.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:26.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:26.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:26.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:30:26.057 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:30:26.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:26.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:26.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:27.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:27.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:27.516 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:30:27.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:30:27.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:27.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:28.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:29.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:29.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:29.217 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:30:29.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:30:29.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:29.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:29.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:30.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:30.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:30.836 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:30:30.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:30:30.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:30.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:32.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:33.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:33.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:33.104 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:30:33.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:30:33.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:33.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:33.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:34.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:34.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:34.938 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:30:34.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:30:34.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:34.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:35.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:36.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:36.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:36.722 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:30:36.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:30:36.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:36.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:37.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:38.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:38.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:38.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:30:38.565 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:30:38.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:38.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:39.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:40.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:40.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:40.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:30:40.592 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:30:40.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:40.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:41.018 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1104937110,v1:127.0.0.1:6819/1104937110] [v2:127.0.0.1:6820/1104937110,v1:127.0.0.1:6821/1104937110] exists,up c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:30:41.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:41.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:41.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:41.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=101144 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:30:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:41.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:30:41.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:30:41.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:30:41.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:30:41.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:30:41.028 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:30:41.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 37b02a8d-a8c0-487b-bbd9-117b387db091' 2024-04-25T07:30:41.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:30:41.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChBipmNW+/AhAA4gS9E/2L08ubJ93VxsybiQ== 2024-04-25T07:30:41.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChBipmNW+/AhAA4gS9E/2L08ubJ93VxsybiQ=="}' 2024-04-25T07:30:41.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37b02a8d-a8c0-487b-bbd9-117b387db091 -i td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:30:41.709 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:30:41.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:30:41.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChBipmNW+/AhAA4gS9E/2L08ubJ93VxsybiQ== --osd-uuid 37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:30:41.764 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.760+0000 7f067aa0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:41.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.796+0000 7f067aa0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:41.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.800+0000 7f067aa0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:41.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.800+0000 7f067aa0f7c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:30:41.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.800+0000 7f067aa0f7c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:30:41.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.800+0000 7f067aa0f7c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:30:41.936 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:41.936+0000 7f067aa0f7c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-04-25T07:30:52.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-04-25T07:30:52.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:30:52.765 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T07:30:52.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:30:52.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:30:53.340 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T07:30:53.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:30:53.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:30:53.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:30:53.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:53.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:30:53.372 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:53.368+0000 7f528de817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:53.404+0000 7f528de817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:53.408+0000 7f528de817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:53.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:30:53.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:53.788 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:30:53.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:30:53.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:53.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:53.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:53.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:53.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:53.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:54.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:54.460 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:54.456+0000 7f528de817c0 -1 Falling back to public interface 2024-04-25T07:30:55.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:55.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:55.274 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:30:55.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:30:55.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:55.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:55.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:56.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:56.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:56.695 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:30:56.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:30:56.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:56.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:57.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:58.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:58.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:58.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:30:58.177 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:30:58.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:58.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:58.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:59.344 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:30:59.344+0000 7f528de817c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:30:59.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:59.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:59.603 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:30:59.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:30:59.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:59.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:00.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:01.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:01.260 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:31:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:31:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:01.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:01.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:02.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:02.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:02.682 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:31:02.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:31:02.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:02.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:03.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:04.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:04.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:04.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:31:04.245 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:31:04.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:04.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:04.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:05.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:05.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:05.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:31:05.706 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:31:05.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:05.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:06.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:07.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:07.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:07.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:31:07.291 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:31:07.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:07.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:07.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:08.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:08.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:08.727 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:31:08.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:31:08.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:08.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:09.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:10.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:10.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:10.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:31:10.236 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:31:10.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:10.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:11.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:11.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:11.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:31:11.700 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:31:11.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:11.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:12.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:13.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:13.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:13.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:31:13.260 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:31:13.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:13.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:13.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:14.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:14.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:14.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:31:14.958 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:31:14.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:14.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:15.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:16.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:16.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:16.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:31:16.740 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:31:16.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:16.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:17.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:18.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:18.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:18.553 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:31:18.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:31:18.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:18.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:19.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:20.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:20.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:20.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:31:20.616 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:31:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:21.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:22.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:22.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:22.383 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:31:22.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:31:22.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:22.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:23.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:24.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:24.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:24.462 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:31:24.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:31:24.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:24.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:31:24.925 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3555018229,v1:127.0.0.1:6827/3555018229] [v2:127.0.0.1:6828/3555018229,v1:127.0.0.1:6829/3555018229] exists,up 37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:31:24.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:31:24.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:31:24.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:31:24.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:31:24.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=103686 2024-04-25T07:31:24.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-04-25T07:31:29.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-04-25T07:31:29.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:31:30.499 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T07:31:30.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:31:31.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:31:31.558 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:31:31.558 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:31:31.559 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_105042 2024-04-25T07:31:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:31:31.559 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:31:32.559 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 26 10 0.0390579 0.0390625 0.274678 0.460407 2024-04-25T07:31:33.559 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 74 58 0.113266 0.1875 0.438442 0.534966 2024-04-25T07:31:34.559 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 111 95 0.123681 0.144531 0.378333 0.473267 2024-04-25T07:31:35.559 INFO:tasks.workunit.client.0.smithi031.stdout: 4 16 169 153 0.149395 0.226562 0.279976 0.410392 2024-04-25T07:31:36.559 INFO:tasks.workunit.client.0.smithi031.stdout: 5 16 219 203 0.158572 0.195312 0.310475 0.382339 2024-04-25T07:31:37.559 INFO:tasks.workunit.client.0.smithi031.stdout: 6 16 269 253 0.164691 0.195312 0.292979 0.370689 2024-04-25T07:31:38.560 INFO:tasks.workunit.client.0.smithi031.stdout: 7 16 320 304 0.169619 0.199219 0.283438 0.358136 2024-04-25T07:31:39.560 INFO:tasks.workunit.client.0.smithi031.stdout: 8 16 379 363 0.177221 0.230469 0.276149 0.346002 2024-04-25T07:31:40.560 INFO:tasks.workunit.client.0.smithi031.stdout: 9 16 432 416 0.18053 0.207031 0.306155 0.341457 2024-04-25T07:31:41.560 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16 489 473 0.18474 0.222656 0.306073 0.335168 2024-04-25T07:31:42.560 INFO:tasks.workunit.client.0.smithi031.stdout: 11 16 540 524 0.186055 0.199219 0.248064 0.332114 2024-04-25T07:31:43.560 INFO:tasks.workunit.client.0.smithi031.stdout: 12 16 596 580 0.188777 0.21875 0.256143 0.327665 2024-04-25T07:31:44.560 INFO:tasks.workunit.client.0.smithi031.stdout: 13 16 645 629 0.188977 0.191406 0.296469 0.328457 2024-04-25T07:31:45.560 INFO:tasks.workunit.client.0.smithi031.stdout: 14 16 694 678 0.189148 0.191406 0.288144 0.325877 2024-04-25T07:31:46.561 INFO:tasks.workunit.client.0.smithi031.stdout: 15 16 751 735 0.19138 0.222656 0.277001 0.322946 2024-04-25T07:31:47.561 INFO:tasks.workunit.client.0.smithi031.stdout: 16 16 806 790 0.192844 0.214844 0.281846 0.321287 2024-04-25T07:31:48.561 INFO:tasks.workunit.client.0.smithi031.stdout: 17 16 854 838 0.192528 0.1875 0.335677 0.321218 2024-04-25T07:31:49.561 INFO:tasks.workunit.client.0.smithi031.stdout: 18 16 908 892 0.193549 0.210938 0.301127 0.320076 2024-04-25T07:31:50.561 INFO:tasks.workunit.client.0.smithi031.stdout: 19 16 965 949 0.19508 0.222656 0.303927 0.318941 2024-04-25T07:31:51.561 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:31:51.564586+0000 min lat: 0.20542 max lat: 1.01163 avg lat: 0.317285 2024-04-25T07:31:51.561 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:31:51.562 INFO:tasks.workunit.client.0.smithi031.stdout: 20 16 1014 998 0.194895 0.191406 0.30059 0.317285 2024-04-25T07:31:52.561 INFO:tasks.workunit.client.0.smithi031.stdout: 21 16 1068 1052 0.195658 0.210938 0.280309 0.317776 2024-04-25T07:31:53.562 INFO:tasks.workunit.client.0.smithi031.stdout: 22 16 1115 1099 0.195108 0.183594 0.273977 0.317527 2024-04-25T07:31:54.562 INFO:tasks.workunit.client.0.smithi031.stdout: 23 16 1167 1151 0.195456 0.203125 0.275144 0.317471 2024-04-25T07:31:55.562 INFO:tasks.workunit.client.0.smithi031.stdout: 24 16 1217 1201 0.195449 0.195312 0.345363 0.318337 2024-04-25T07:31:56.562 INFO:tasks.workunit.client.0.smithi031.stdout: 25 16 1266 1250 0.195286 0.191406 0.333307 0.317619 2024-04-25T07:31:57.562 INFO:tasks.workunit.client.0.smithi031.stdout: 26 16 1325 1309 0.196638 0.230469 0.257925 0.316462 2024-04-25T07:31:58.562 INFO:tasks.workunit.client.0.smithi031.stdout: 27 16 1375 1359 0.196588 0.195312 0.299363 0.315429 2024-04-25T07:31:59.562 INFO:tasks.workunit.client.0.smithi031.stdout: 28 16 1439 1423 0.198494 0.25 0.293057 0.313387 2024-04-25T07:32:00.562 INFO:tasks.workunit.client.0.smithi031.stdout: 29 16 1487 1471 0.198114 0.1875 0.325153 0.313487 2024-04-25T07:32:01.563 INFO:tasks.workunit.client.0.smithi031.stdout: 30 16 1551 1535 0.199842 0.25 0.294249 0.311773 2024-04-25T07:32:02.563 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 30.3332 2024-04-25T07:32:02.563 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 1551 2024-04-25T07:32:02.563 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4096 2024-04-25T07:32:02.563 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4096 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.199735 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0371725 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.25 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.0390625 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 51 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 9.51617 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 64 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 10 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.311889 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0739283 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 1.01163 2024-04-25T07:32:02.564 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.20542 2024-04-25T07:32:02.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-04-25T07:32:02.698 INFO:tasks.workunit.client.0.smithi031.stdout:after bench 2024-04-25T07:32:02.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 96010 2024-04-25T07:32:02.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-04-25T07:32:03.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 96010 2024-04-25T07:32:03.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (96010) - No such process 2024-04-25T07:32:03.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-04-25T07:32:04.494 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:32:04.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98578 2024-04-25T07:32:04.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-04-25T07:32:05.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98578 2024-04-25T07:32:05.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (98578) - No such process 2024-04-25T07:32:05.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-04-25T07:32:06.145 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:32:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 101144 2024-04-25T07:32:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-04-25T07:32:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 101144 2024-04-25T07:32:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (101144) - No such process 2024-04-25T07:32:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-04-25T07:32:07.620 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:32:07.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 103686 2024-04-25T07:32:07.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-04-25T07:32:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 103686 2024-04-25T07:32:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (103686) - No such process 2024-04-25T07:32:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-04-25T07:32:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:osd.3 is already down. 2024-04-25T07:32:09.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:32:11.017 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:11.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:32:12.845 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:12.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:32:14.681 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:14.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:32:16.513 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:16.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:32:16.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:32:16.577 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:17.469 INFO:tasks.workunit.client.0.smithi031.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:32:17.470 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:32:17.470 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0x100000000 : using 0x804da000(2.0 GiB) 2024-04-25T07:32:17.470 INFO:tasks.workunit.client.0.smithi031.stdout:Expanding DB/WAL... 2024-04-25T07:32:17.470 INFO:tasks.workunit.client.0.smithi031.stdout:0 : expanding to 0x536870912 2024-04-25T07:32:17.504 INFO:tasks.workunit.client.0.smithi031.stdout:0 : size label updated to 536870912 2024-04-25T07:32:17.504 INFO:tasks.workunit.client.0.smithi031.stdout:1 : expanding to 0x1073741824 2024-04-25T07:32:17.529 INFO:tasks.workunit.client.0.smithi031.stdout:1 : size label updated to 1073741824 2024-04-25T07:32:17.529 INFO:tasks.workunit.client.0.smithi031.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-04-25T07:32:17.738 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:17.738+0000 7ff2c08d9ac0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:32:17.738 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:17.738+0000 7ff2c08d9ac0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-04-25T07:32:23.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-04-25T07:32:23.080 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:23.984 INFO:tasks.workunit.client.0.smithi031.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:32:23.984 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:32:23.984 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0x101000000 : using 0x814cd000(2.0 GiB) 2024-04-25T07:32:23.984 INFO:tasks.workunit.client.0.smithi031.stdout:Expanding DB/WAL... 2024-04-25T07:32:23.984 INFO:tasks.workunit.client.0.smithi031.stdout:0 : expanding to 0x536870912 2024-04-25T07:32:24.019 INFO:tasks.workunit.client.0.smithi031.stdout:0 : size label updated to 536870912 2024-04-25T07:32:24.019 INFO:tasks.workunit.client.0.smithi031.stdout:1 : expanding to 0x1073741824 2024-04-25T07:32:24.052 INFO:tasks.workunit.client.0.smithi031.stdout:1 : size label updated to 1073741824 2024-04-25T07:32:24.052 INFO:tasks.workunit.client.0.smithi031.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-04-25T07:32:24.329 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:24.330+0000 7f6ca798eac0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:32:24.329 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:24.330+0000 7f6ca798eac0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:32:29.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-04-25T07:32:29.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-04-25T07:32:29.753 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:30.650 INFO:tasks.workunit.client.0.smithi031.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:32:30.651 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:32:30.651 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0x100020000 : using 0x80507000(2.0 GiB) 2024-04-25T07:32:30.651 INFO:tasks.workunit.client.0.smithi031.stdout:Expanding DB/WAL... 2024-04-25T07:32:30.651 INFO:tasks.workunit.client.0.smithi031.stdout:0 : expanding to 0x536870912 2024-04-25T07:32:30.676 INFO:tasks.workunit.client.0.smithi031.stdout:0 : size label updated to 536870912 2024-04-25T07:32:30.676 INFO:tasks.workunit.client.0.smithi031.stdout:1 : expanding to 0x1073741824 2024-04-25T07:32:30.709 INFO:tasks.workunit.client.0.smithi031.stdout:1 : size label updated to 1073741824 2024-04-25T07:32:30.710 INFO:tasks.workunit.client.0.smithi031.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-04-25T07:32:30.936 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:30.938+0000 7f14ddae8ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:32:30.936 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:30.938+0000 7f14ddae8ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:32:36.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-04-25T07:32:36.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-04-25T07:32:36.294 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:32:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:32:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0xfff00000 : using 0x803bc000(2.0 GiB) 2024-04-25T07:32:37.207 INFO:tasks.workunit.client.0.smithi031.stdout:Expanding DB/WAL... 2024-04-25T07:32:37.208 INFO:tasks.workunit.client.0.smithi031.stdout:0 : expanding to 0x536870912 2024-04-25T07:32:37.240 INFO:tasks.workunit.client.0.smithi031.stdout:0 : size label updated to 536870912 2024-04-25T07:32:37.240 INFO:tasks.workunit.client.0.smithi031.stdout:1 : expanding to 0x1073741824 2024-04-25T07:32:37.274 INFO:tasks.workunit.client.0.smithi031.stdout:1 : size label updated to 1073741824 2024-04-25T07:32:37.274 INFO:tasks.workunit.client.0.smithi031.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-04-25T07:32:37.492 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:37.494+0000 7fa79281bac0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:32:37.492 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:32:37.494+0000 7fa79281bac0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:32:42.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:32:44.729 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:44.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:32:46.565 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:46.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:32:48.397 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:32:50.233 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:32:50.282 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:52.071 INFO:tasks.workunit.client.0.smithi031.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-04-25T07:32:52.071 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-04-25T07:32:52.071 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0x100000000 : using 0x4da000(4.9 MiB) 2024-04-25T07:32:52.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-04-25T07:32:52.100 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:53.897 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:32:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:32:55.217 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:55.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:32:55.267 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:32:57.297 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:32:57.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:32:58.613 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:32:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-04-25T07:32:58.660 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:33:00.441 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:33:00.442 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:33:00.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:33:01.257 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:33:01.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-04-25T07:33:01.299 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:33:01.299 INFO:tasks.workunit.client.0.smithi031.stderr:Migrate to WAL device isn't supported. 2024-04-25T07:33:01.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:33:01.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:33:03.117 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:33:03.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:33:03.169 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:33:05.209 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:33:05.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:33:06.529 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:33:06.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-04-25T07:33:06.580 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:33:07.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:33:09.181 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:33:09.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:09.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:33:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:33:09.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:33:09.210 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:33:09.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:33:09.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:09.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:33:09.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:33:09.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:09.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:09.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:09.245 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:09.241+0000 7f0b34ebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:09.247 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:09.245+0000 7f0b34ebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:09.249 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:09.245+0000 7f0b34ebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:09.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:33:09.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:09.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:33:09.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:09.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:09.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:09.651 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:33:09.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:09.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:09.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:10.036 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:10.033+0000 7f0b34ebb7c0 -1 Falling back to public interface 2024-04-25T07:33:10.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:11.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:11.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:11.087 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:33:11.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:11.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:11.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:11.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:12.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:12.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:12.521 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:33:12.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:12.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:12.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:12.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:13.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:13.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:13.940 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:33:13.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:13.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:13.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:13.969 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:13.965+0000 7f0b34ebb7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:33:14.382 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 40 up_thru 22 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1732661318,v1:127.0.0.1:6803/1732661318] [v2:127.0.0.1:6804/1732661318,v1:127.0.0.1:6805/1732661318] exists,up 937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:33:14.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:14.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:14.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:14.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=117118 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:33:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:14.387 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:33:14.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:14.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:33:14.390 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:33:14.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:33:14.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:14.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:33:14.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:33:14.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:14.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:14.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:14.423 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:14.421+0000 7fb5b751e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:14.488 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:14.485+0000 7fb5b751e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:14.489 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:14.489+0000 7fb5b751e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:14.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:33:14.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:14.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:15.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:15.279 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:15.277+0000 7fb5b751e7c0 -1 Falling back to public interface 2024-04-25T07:33:16.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:16.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:16.207 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:33:16.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:16.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:16.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:16.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:17.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:17.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:17.641 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:33:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:18.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:19.080 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:19.077+0000 7fb5b751e7c0 -1 osd.1 32 log_to_monitors true 2024-04-25T07:33:19.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:19.083 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:33:19.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:19.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:19.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:19.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:19.685 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 43 up_thru 30 down_at 33 last_clean_interval [10,32) [v2:127.0.0.1:6810/1337544441,v1:127.0.0.1:6811/1337544441] [v2:127.0.0.1:6812/1337544441,v1:127.0.0.1:6813/1337544441] exists,up cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:33:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:19.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:33:19.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=118059 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:33:19.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:19.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:19.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:33:19.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:19.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:19.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:19.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:33:19.692 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:33:19.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:33:19.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:19.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:33:19.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:33:19.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:19.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:19.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:19.723 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:19.721+0000 7fd8458ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:19.817 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:19.817+0000 7fd8458ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:19.819 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:19.817+0000 7fd8458ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:20.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:20.329 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:33:20.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:20.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:20.367 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:20.365+0000 7fd8458ce7c0 -1 Falling back to public interface 2024-04-25T07:33:20.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:21.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:21.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:21.995 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:33:21.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:21.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:21.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:23.397 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:23.393+0000 7fd8458ce7c0 -1 osd.2 35 log_to_monitors true 2024-04-25T07:33:23.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:23.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:23.429 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:33:23.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:23.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:23.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:24.031 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 47 up_thru 33 down_at 36 last_clean_interval [15,35) [v2:127.0.0.1:6818/543214475,v1:127.0.0.1:6819/543214475] [v2:127.0.0.1:6820/543214475,v1:127.0.0.1:6821/543214475] exists,up c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:33:24.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:24.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:24.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:24.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=119013 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:24.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:24.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:33:24.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:33:24.039 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T07:33:24.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:33:24.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:24.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:33:24.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:33:24.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:24.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:24.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:24.063 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:24.061+0000 7f0ce6cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:24.271 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:24.269+0000 7f0ce6cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:24.273 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:24.269+0000 7f0ce6cef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:24.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:33:24.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:24.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:33:24.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:24.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:24.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:24.849 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:33:24.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:24.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:24.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:25.063 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:25.061+0000 7f0ce6cef7c0 -1 Falling back to public interface 2024-04-25T07:33:25.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:26.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:26.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:26.419 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:33:26.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:26.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:26.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:26.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:27.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:27.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:27.847 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:33:27.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:27.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:27.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:28.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:29.090 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:33:29.089+0000 7f0ce6cef7c0 -1 osd.3 37 log_to_monitors true 2024-04-25T07:33:29.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:29.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:29.312 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:33:29.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:29.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:29.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:30.062 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 51 up_thru 36 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/529393004,v1:127.0.0.1:6827/529393004] [v2:127.0.0.1:6828/529393004,v1:127.0.0.1:6829/529393004] exists,up 37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:33:30.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:30.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:30.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:30.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:33:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=119922 2024-04-25T07:33:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-04-25T07:33:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:33:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:33:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:33:30.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:33:30.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:33:30.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:33:30.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:33:30.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:33:30.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:33:30.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:33:30.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:33:30.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:33:30.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:33:30.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:33:30.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:33:30.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:33:30.845 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:33:30.845 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T07:33:30.845 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T07:33:30.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:33:30.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:30.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:33:31.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691846 2024-04-25T07:33:31.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691846 2024-04-25T07:33:31.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691846' 2024-04-25T07:33:31.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:31.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:33:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=184683593733 2024-04-25T07:33:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 184683593733 2024-04-25T07:33:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691846 1-184683593733' 2024-04-25T07:33:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:31.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:33:31.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=201863462916 2024-04-25T07:33:31.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 201863462916 2024-04-25T07:33:31.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691846 1-184683593733 2-201863462916' 2024-04-25T07:33:31.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:31.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:33:31.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=219043332099 2024-04-25T07:33:31.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 219043332099 2024-04-25T07:33:31.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-171798691846 1-184683593733 2-201863462916 3-219043332099' 2024-04-25T07:33:31.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:31.663 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-171798691846 2024-04-25T07:33:31.663 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:31.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:33:31.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-171798691846 2024-04-25T07:33:31.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:31.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691846 2024-04-25T07:33:31.666 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 171798691846 2024-04-25T07:33:31.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 171798691846' 2024-04-25T07:33:31.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:33:32.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691846 -lt 171798691846 2024-04-25T07:33:32.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:32.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-184683593733 2024-04-25T07:33:32.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:32.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:33:32.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-184683593733 2024-04-25T07:33:32.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:32.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=184683593733 2024-04-25T07:33:32.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 184683593733' 2024-04-25T07:33:32.181 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 184683593733 2024-04-25T07:33:32.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:33:32.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593733 -lt 184683593733 2024-04-25T07:33:32.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:32.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-201863462916 2024-04-25T07:33:32.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:32.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:33:32.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-201863462916 2024-04-25T07:33:32.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:32.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=201863462916 2024-04-25T07:33:32.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 201863462916' 2024-04-25T07:33:32.641 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 201863462916 2024-04-25T07:33:32.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:33:33.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 201863462916 -lt 201863462916 2024-04-25T07:33:33.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:33.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-219043332099 2024-04-25T07:33:33.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:33.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:33:33.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-219043332099 2024-04-25T07:33:33.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:33.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=219043332099 2024-04-25T07:33:33.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 219043332099' 2024-04-25T07:33:33.123 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 219043332099 2024-04-25T07:33:33.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:33:33.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 219043332099 -lt 219043332099 2024-04-25T07:33:33.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:33:33.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:33.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:34.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:33:34.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:33:34.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:33:34.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:33:34.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:33:34.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:33:34.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:33:34.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:33:34.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:33:34.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:33:34.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:34.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:35.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:33:35.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:33:35.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:33:35.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:33:35.148 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:33:35.148 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:33:35.148 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_121188 2024-04-25T07:33:35.148 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:33:35.149 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:33:36.148 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 60 44 0.171861 0.171875 0.198465 0.308033 2024-04-25T07:33:37.148 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 111 95 0.185525 0.199219 0.365565 0.310333 2024-04-25T07:33:38.148 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 163 147 0.191384 0.203125 0.134141 0.304042 2024-04-25T07:33:39.149 INFO:tasks.workunit.client.0.smithi031.stdout: 4 16 216 200 0.195288 0.207031 0.275523 0.306103 2024-04-25T07:33:40.149 INFO:tasks.workunit.client.0.smithi031.stdout: 5 16 276 260 0.2031 0.234375 0.331984 0.30034 2024-04-25T07:33:41.149 INFO:tasks.workunit.client.0.smithi031.stdout: 6 16 328 312 0.203099 0.203125 0.313458 0.301826 2024-04-25T07:33:42.149 INFO:tasks.workunit.client.0.smithi031.stdout: 7 16 385 369 0.205888 0.222656 0.366486 0.299185 2024-04-25T07:33:43.149 INFO:tasks.workunit.client.0.smithi031.stdout: 8 16 429 413 0.201633 0.171875 0.27737 0.301623 2024-04-25T07:33:44.149 INFO:tasks.workunit.client.0.smithi031.stdout: 9 16 484 468 0.203098 0.214844 0.292592 0.302425 2024-04-25T07:33:45.149 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16 546 530 0.207003 0.242188 0.191709 0.297777 2024-04-25T07:33:46.150 INFO:tasks.workunit.client.0.smithi031.stdout: 11 16 597 581 0.206293 0.199219 0.34219 0.299384 2024-04-25T07:33:47.150 INFO:tasks.workunit.client.0.smithi031.stdout: 12 16 651 635 0.206677 0.210938 0.364549 0.29916 2024-04-25T07:33:48.150 INFO:tasks.workunit.client.0.smithi031.stdout: 13 16 694 678 0.203698 0.167969 0.274655 0.29877 2024-04-25T07:33:49.150 INFO:tasks.workunit.client.0.smithi031.stdout: 14 16 750 734 0.20477 0.21875 0.283546 0.30318 2024-04-25T07:33:50.150 INFO:tasks.workunit.client.0.smithi031.stdout: 15 16 807 791 0.205961 0.222656 0.237922 0.300912 2024-04-25T07:33:51.150 INFO:tasks.workunit.client.0.smithi031.stdout: 16 16 853 837 0.204317 0.179688 0.151052 0.302846 2024-04-25T07:33:52.150 INFO:tasks.workunit.client.0.smithi031.stdout: 17 16 907 891 0.204705 0.210938 0.299888 0.302571 2024-04-25T07:33:53.150 INFO:tasks.workunit.client.0.smithi031.stdout: 18 16 956 940 0.203964 0.191406 0.277456 0.304265 2024-04-25T07:33:54.151 INFO:tasks.workunit.client.0.smithi031.stdout: 19 16 1009 993 0.204124 0.207031 0.151768 0.304776 2024-04-25T07:33:55.151 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:33:55.154165+0000 min lat: 0.1073 max lat: 0.596394 avg lat: 0.301403 2024-04-25T07:33:55.151 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:33:55.151 INFO:tasks.workunit.client.0.smithi031.stdout: 20 16 1072 1056 0.206221 0.246094 0.232279 0.301403 2024-04-25T07:33:56.151 INFO:tasks.workunit.client.0.smithi031.stdout: 21 16 1145 1129 0.209977 0.285156 0.212364 0.295202 2024-04-25T07:33:57.151 INFO:tasks.workunit.client.0.smithi031.stdout: 22 16 1214 1198 0.212683 0.269531 0.286784 0.292122 2024-04-25T07:33:58.151 INFO:tasks.workunit.client.0.smithi031.stdout: 23 16 1278 1262 0.214304 0.25 0.231887 0.290363 2024-04-25T07:33:59.152 INFO:tasks.workunit.client.0.smithi031.stdout: 24 16 1324 1308 0.21286 0.179688 0.307722 0.291652 2024-04-25T07:34:00.152 INFO:tasks.workunit.client.0.smithi031.stdout: 25 16 1380 1364 0.213095 0.21875 0.287228 0.291305 2024-04-25T07:34:01.152 INFO:tasks.workunit.client.0.smithi031.stdout: 26 16 1446 1430 0.214813 0.257812 0.226017 0.289489 2024-04-25T07:34:02.152 INFO:tasks.workunit.client.0.smithi031.stdout: 27 16 1512 1496 0.216404 0.257812 0.260296 0.287467 2024-04-25T07:34:03.152 INFO:tasks.workunit.client.0.smithi031.stdout: 28 16 1576 1560 0.217603 0.25 0.276552 0.286127 2024-04-25T07:34:04.152 INFO:tasks.workunit.client.0.smithi031.stdout: 29 16 1639 1623 0.218584 0.246094 0.250122 0.284478 2024-04-25T07:34:05.152 INFO:tasks.workunit.client.0.smithi031.stdout: 30 16 1703 1687 0.21963 0.25 0.296279 0.283196 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 30.2104 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 1703 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4096 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4096 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.2202 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.030906 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.285156 2024-04-25T07:34:06.153 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.167969 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 56 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 7.91194 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 73 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 43 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.283128 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0708291 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.596394 2024-04-25T07:34:06.154 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.1073 2024-04-25T07:34:06.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 117118 2024-04-25T07:34:06.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-04-25T07:34:07.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 117118 2024-04-25T07:34:07.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (117118) - No such process 2024-04-25T07:34:07.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-04-25T07:34:07.859 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:34:07.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 118059 2024-04-25T07:34:07.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-04-25T07:34:08.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 118059 2024-04-25T07:34:08.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (118059) - No such process 2024-04-25T07:34:08.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-04-25T07:34:09.369 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:34:09.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 119013 2024-04-25T07:34:09.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-04-25T07:34:10.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 119013 2024-04-25T07:34:10.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (119013) - No such process 2024-04-25T07:34:10.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-04-25T07:34:10.858 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:34:10.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 119922 2024-04-25T07:34:10.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-04-25T07:34:11.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 119922 2024-04-25T07:34:11.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (119922) - No such process 2024-04-25T07:34:11.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-04-25T07:34:12.569 INFO:tasks.workunit.client.0.smithi031.stderr:osd.3 is already down. 2024-04-25T07:34:12.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:34:13.893 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:13.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-04-25T07:34:14.220 INFO:tasks.workunit.client.0.smithi031.stderr:512+0 records in 2024-04-25T07:34:14.220 INFO:tasks.workunit.client.0.smithi031.stderr:512+0 records out 2024-04-25T07:34:14.220 INFO:tasks.workunit.client.0.smithi031.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.304516 s, 1.8 GB/s 2024-04-25T07:34:14.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-04-25T07:34:14.244 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:18.777 INFO:tasks.workunit.client.0.smithi031.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:34:18.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:34:20.601 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:20.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:34:21.929 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:21.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-04-25T07:34:22.521 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records in 2024-04-25T07:34:22.521 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records out 2024-04-25T07:34:22.522 INFO:tasks.workunit.client.0.smithi031.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.574302 s, 1.9 GB/s 2024-04-25T07:34:22.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-04-25T07:34:22.546 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:30.581 INFO:tasks.workunit.client.0.smithi031.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-04-25T07:34:30.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-04-25T07:34:30.617 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:32.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:34:34.225 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:34.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:34:35.053 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:35.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-04-25T07:34:35.091 INFO:tasks.workunit.client.0.smithi031.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:34:35.091 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:35.881 INFO:tasks.workunit.client.0.smithi031.stdout:DB device added 2024-04-25T07:34:35.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-04-25T07:34:35.914 INFO:tasks.workunit.client.0.smithi031.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:34:35.914 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:37.197 INFO:tasks.workunit.client.0.smithi031.stdout:WAL device added 2024-04-25T07:34:37.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-04-25T07:34:37.232 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:39.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:34:41.333 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:41.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:34:42.661 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:42.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-04-25T07:34:42.706 INFO:tasks.workunit.client.0.smithi031.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:34:42.706 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:43.989 INFO:tasks.workunit.client.0.smithi031.stdout:WAL device added 2024-04-25T07:34:43.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:34:44.022 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:34:46.073 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-04-25T07:34:46.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:34:47.393 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:34:47.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:34:47.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:34:47.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:34:47.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:34:47.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:34:47.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:34:47.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:34:47.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:34:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:34:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:34:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:34:47.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:34:47.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:34:47.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:34:47.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:34:47.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:34:47.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:34:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:34:47.418 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:34:47.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:34:47.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:34:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:34:47.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:34:47.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:34:47.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:34:47.451 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:47.452+0000 7fe257dc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:47.454 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:47.456+0000 7fe257dc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:47.456 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:47.456+0000 7fe257dc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:47.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:34:47.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:34:47.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:34:47.850 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:34:47.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:34:47.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:34:47.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:47.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:34:47.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:47.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:48.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:48.492 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:48.492+0000 7fe257dc97c0 -1 Falling back to public interface 2024-04-25T07:34:49.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:49.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:49.280 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:34:49.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:34:49.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:49.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:49.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:50.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:50.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:50.712 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:34:50.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:34:50.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:50.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:51.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:52.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:52.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:52.144 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:34:52.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:34:52.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:52.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:52.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:53.317 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:53.316+0000 7fe257dc97c0 -1 osd.0 53 log_to_monitors true 2024-04-25T07:34:53.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:53.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:53.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:34:53.609 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:34:53.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:53.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:54.132 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 64 up_thru 64 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6802/4162051106,v1:127.0.0.1:6803/4162051106] [v2:127.0.0.1:6804/4162051106,v1:127.0.0.1:6805/4162051106] exists,up 937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:34:54.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:34:54.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:34:54.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:34:54.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=127296 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:34:54.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:34:54.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:34:54.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:34:54.140 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:34:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:34:54.168 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:54.168+0000 7f29fde5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:54.192 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:54.192+0000 7f29fde5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:54.194 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:54.196+0000 7f29fde5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:54.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:34:54.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:34:54.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:34:54.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:34:54.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:34:54.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:54.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:34:54.562 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:34:54.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:54.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:54.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:55.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:34:55.232+0000 7f29fde5c7c0 -1 Falling back to public interface 2024-04-25T07:34:55.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:55.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:56.000 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:34:56.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:34:56.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:56.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:56.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:57.433 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:34:57.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:34:57.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:57.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:57.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:58.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:58.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:58.856 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:34:58.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:34:58.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:58.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:59.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:00.000 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:00.000+0000 7f29fde5c7c0 -1 osd.1 55 log_to_monitors true 2024-04-25T07:35:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:00.314 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:35:00.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:35:00.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:00.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:00.904 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 57 last_clean_interval [43,55) [v2:127.0.0.1:6810/2625246175,v1:127.0.0.1:6811/2625246175] [v2:127.0.0.1:6812/2625246175,v1:127.0.0.1:6813/2625246175] exists,up cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:35:00.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:35:00.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:35:00.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:35:00.905 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:35:00.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=128265 2024-04-25T07:35:00.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:35:00.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:35:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:35:00.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:35:00.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:35:00.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:35:00.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:35:00.913 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:35:00.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:35:00.913 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:35:00.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:35:00.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:35:00.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:35:00.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:35:00.945 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:00.948+0000 7f57cfa117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:01.033 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:01.036+0000 7f57cfa117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:01.035 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:01.036+0000 7f57cfa117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:01.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:35:01.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:35:01.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:35:01.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:35:01.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:35:01.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:01.390 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:35:01.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:35:01.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:01.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:01.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:02.072 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:02.072+0000 7f57cfa117c0 -1 Falling back to public interface 2024-04-25T07:35:02.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:02.880 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:35:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:35:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:03.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:04.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:04.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:04.351 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:35:04.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:35:04.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:04.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:04.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:05.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:05.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:05.845 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:35:05.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:35:05.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:05.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:06.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:07.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:07.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:07.375 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:35:07.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:35:07.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:07.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:07.577 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:07.580+0000 7f57cfa117c0 -1 osd.2 59 log_to_monitors true 2024-04-25T07:35:07.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:08.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:08.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:08.820 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:35:08.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:35:08.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:08.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:09.623 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 70 up_thru 70 down_at 60 last_clean_interval [47,59) [v2:127.0.0.1:6818/1963371964,v1:127.0.0.1:6819/1963371964] [v2:127.0.0.1:6820/1963371964,v1:127.0.0.1:6821/1963371964] exists,up c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:35:09.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:35:09.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:35:09.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:35:09.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:35:09.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=129245 2024-04-25T07:35:09.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:35:09.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:35:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:35:09.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:35:09.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:35:09.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:35:09.630 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:35:09.630 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:35:09.630 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:35:09.630 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:35:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:35:09.632 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T07:35:09.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:35:09.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:35:09.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:35:09.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:35:09.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:35:09.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:35:09.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:35:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:09.664+0000 7f00820a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:09.723 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:09.724+0000 7f00820a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:09.724 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:09.724+0000 7f00820a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:10.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:35:10.173 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:35:10.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:10.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:35:10.511 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:10.511+0000 7f00820a67c0 -1 Falling back to public interface 2024-04-25T07:35:10.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:11.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:11.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:11.630 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:35:11.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:35:11.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:11.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:35:12.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:13.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:13.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:13.056 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:35:13.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:35:13.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:13.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:35:13.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:14.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:14.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:14.546 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:35:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:35:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:35:14.870 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:35:14.871+0000 7f00820a67c0 -1 osd.3 61 log_to_monitors true 2024-04-25T07:35:14.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:15.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:15.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:15.976 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:35:15.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:35:15.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:15.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:35:16.621 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 74 up_thru 60 down_at 62 last_clean_interval [51,61) [v2:127.0.0.1:6826/2725555891,v1:127.0.0.1:6827/2725555891] [v2:127.0.0.1:6828/2725555891,v1:127.0.0.1:6829/2725555891] exists,up 37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:35:16.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:35:16.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:35:16.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:35:16.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:35:16.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=130233 2024-04-25T07:35:16.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:35:16.657 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:35:16.657 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:35:16.657 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_131182 2024-04-25T07:35:16.657 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:35:16.657 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:35:17.657 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 31 15 0.0585877 0.0585938 0.226202 0.412979 2024-04-25T07:35:18.657 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 68 52 0.101552 0.144531 0.333333 0.573292 2024-04-25T07:35:19.658 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 119 103 0.134098 0.199219 0.214422 0.444917 2024-04-25T07:35:20.658 INFO:tasks.workunit.client.0.smithi031.stdout: 4 16 162 146 0.142559 0.167969 0.333233 0.411177 2024-04-25T07:35:21.658 INFO:tasks.workunit.client.0.smithi031.stdout: 5 16 211 195 0.152324 0.191406 0.472018 0.396297 2024-04-25T07:35:22.658 INFO:tasks.workunit.client.0.smithi031.stdout: 6 16 259 243 0.158184 0.1875 0.376153 0.384473 2024-04-25T07:35:23.658 INFO:tasks.workunit.client.0.smithi031.stdout: 7 16 303 287 0.160136 0.171875 0.35234 0.379117 2024-04-25T07:35:24.658 INFO:tasks.workunit.client.0.smithi031.stdout: 8 16 355 339 0.165506 0.203125 0.326509 0.369752 2024-04-25T07:35:25.658 INFO:tasks.workunit.client.0.smithi031.stdout: 9 16 406 390 0.169248 0.199219 0.336632 0.36306 2024-04-25T07:35:26.658 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16 445 429 0.167556 0.152344 0.524443 0.361214 2024-04-25T07:35:27.659 INFO:tasks.workunit.client.0.smithi031.stdout: 11 16 493 477 0.169367 0.1875 0.343312 0.364013 2024-04-25T07:35:28.659 INFO:tasks.workunit.client.0.smithi031.stdout: 12 16 541 525 0.170876 0.1875 0.383766 0.360316 2024-04-25T07:35:29.659 INFO:tasks.workunit.client.0.smithi031.stdout: 13 16 591 575 0.172754 0.195312 0.217868 0.358397 2024-04-25T07:35:30.659 INFO:tasks.workunit.client.0.smithi031.stdout: 14 16 634 618 0.172411 0.167969 0.383083 0.357434 2024-04-25T07:35:31.659 INFO:tasks.workunit.client.0.smithi031.stdout: 15 16 688 672 0.174977 0.210938 0.361903 0.354844 2024-04-25T07:35:32.659 INFO:tasks.workunit.client.0.smithi031.stdout: 16 16 731 715 0.174538 0.167969 0.226027 0.355157 2024-04-25T07:35:33.659 INFO:tasks.workunit.client.0.smithi031.stdout: 17 16 778 762 0.175069 0.183594 0.361286 0.354264 2024-04-25T07:35:34.659 INFO:tasks.workunit.client.0.smithi031.stdout: 18 16 823 807 0.175107 0.175781 0.182153 0.352522 2024-04-25T07:35:35.660 INFO:tasks.workunit.client.0.smithi031.stdout: 19 16 875 859 0.17658 0.203125 0.204073 0.351899 2024-04-25T07:35:36.660 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:35:36.663124+0000 min lat: 0.17593 max lat: 1.59935 avg lat: 0.349044 2024-04-25T07:35:36.660 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:35:36.660 INFO:tasks.workunit.client.0.smithi031.stdout: 20 16 927 911 0.177906 0.203125 0.291824 0.349044 2024-04-25T07:35:37.660 INFO:tasks.workunit.client.0.smithi031.stdout: 21 16 978 962 0.178919 0.199219 0.187805 0.347273 2024-04-25T07:35:38.660 INFO:tasks.workunit.client.0.smithi031.stdout: 22 16 1027 1011 0.179486 0.191406 0.223079 0.346037 2024-04-25T07:35:39.660 INFO:tasks.workunit.client.0.smithi031.stdout: 23 16 1073 1057 0.179493 0.179688 0.302613 0.345028 2024-04-25T07:35:40.660 INFO:tasks.workunit.client.0.smithi031.stdout: 24 16 1121 1105 0.179826 0.1875 0.356259 0.34504 2024-04-25T07:35:41.660 INFO:tasks.workunit.client.0.smithi031.stdout: 25 16 1171 1155 0.180444 0.195312 0.350352 0.344527 2024-04-25T07:35:42.661 INFO:tasks.workunit.client.0.smithi031.stdout: 26 16 1207 1191 0.178912 0.140625 0.223391 0.346425 2024-04-25T07:35:43.661 INFO:tasks.workunit.client.0.smithi031.stdout: 27 16 1260 1244 0.179952 0.207031 0.208965 0.345612 2024-04-25T07:35:44.661 INFO:tasks.workunit.client.0.smithi031.stdout: 28 16 1312 1296 0.180779 0.203125 0.309885 0.344502 2024-04-25T07:35:45.661 INFO:tasks.workunit.client.0.smithi031.stdout: 29 16 1356 1340 0.180471 0.171875 0.246481 0.344586 2024-04-25T07:35:46.661 INFO:tasks.workunit.client.0.smithi031.stdout: 30 16 1397 1381 0.179793 0.160156 0.399784 0.345144 2024-04-25T07:35:47.661 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 30.2665 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 1397 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4096 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4096 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.180299 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0294734 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.210938 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.0585938 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 46 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 7.54519 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 54 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 15 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.345707 2024-04-25T07:35:47.662 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.112396 2024-04-25T07:35:47.663 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 1.59935 2024-04-25T07:35:47.663 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.17593 2024-04-25T07:35:47.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 127296 2024-04-25T07:35:47.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-04-25T07:35:48.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 127296 2024-04-25T07:35:48.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (127296) - No such process 2024-04-25T07:35:48.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-04-25T07:35:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:35:49.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 128265 2024-04-25T07:35:49.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-04-25T07:35:50.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 128265 2024-04-25T07:35:50.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (128265) - No such process 2024-04-25T07:35:50.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-04-25T07:35:51.008 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T07:35:51.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 129245 2024-04-25T07:35:51.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-04-25T07:35:52.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 129245 2024-04-25T07:35:52.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (129245) - No such process 2024-04-25T07:35:52.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-04-25T07:35:52.469 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T07:35:52.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 130233 2024-04-25T07:35:52.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-04-25T07:35:53.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 130233 2024-04-25T07:35:53.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (130233) - No such process 2024-04-25T07:35:53.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-04-25T07:35:54.067 INFO:tasks.workunit.client.0.smithi031.stderr:osd.3 is already down. 2024-04-25T07:35:54.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:35:55.885 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:35:55.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-04-25T07:35:56.487 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records in 2024-04-25T07:35:56.487 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records out 2024-04-25T07:35:56.487 INFO:tasks.workunit.client.0.smithi031.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.575704 s, 1.9 GB/s 2024-04-25T07:35:56.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-04-25T07:35:56.512 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:06.053 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:36:06.054 INFO:tasks.workunit.client.0.smithi031.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-04-25T07:36:06.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:36:07.869 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:07.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-04-25T07:36:08.194 INFO:tasks.workunit.client.0.smithi031.stderr:512+0 records in 2024-04-25T07:36:08.194 INFO:tasks.workunit.client.0.smithi031.stderr:512+0 records out 2024-04-25T07:36:08.194 INFO:tasks.workunit.client.0.smithi031.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.303785 s, 1.8 GB/s 2024-04-25T07:36:08.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-04-25T07:36:08.220 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:13.513 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:36:13.514 INFO:tasks.workunit.client.0.smithi031.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-04-25T07:36:13.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:36:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:36:15.401 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:15.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:36:17.233 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:17.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-04-25T07:36:17.832 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records in 2024-04-25T07:36:17.832 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records out 2024-04-25T07:36:17.833 INFO:tasks.workunit.client.0.smithi031.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.580608 s, 1.8 GB/s 2024-04-25T07:36:17.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-04-25T07:36:17.856 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:26.893 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:36:26.894 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-04-25T07:36:26.894 INFO:tasks.workunit.client.0.smithi031.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:36:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-04-25T07:36:27.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:36:28.345 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:28.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:36:28.393 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:29.925 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:36:29.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:36:30.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:36:30.869 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:30.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-04-25T07:36:32.701 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:32.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:36:33.308 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records in 2024-04-25T07:36:33.308 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records out 2024-04-25T07:36:33.309 INFO:tasks.workunit.client.0.smithi031.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.580251 s, 1.9 GB/s 2024-04-25T07:36:33.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:36:33.333 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:33.333 INFO:tasks.workunit.client.0.smithi031.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-04-25T07:36:33.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:36:33.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:36:35.145 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:35.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:36:42.892 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records in 2024-04-25T07:36:42.893 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records out 2024-04-25T07:36:42.893 INFO:tasks.workunit.client.0.smithi031.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.59006 s, 141 MB/s 2024-04-25T07:36:42.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:36:42.920 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:45.457 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:36:45.457 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:36:45.458 INFO:tasks.workunit.client.0.smithi031.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-04-25T07:36:45.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:36:46.773 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:46.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-04-25T07:36:47.370 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records in 2024-04-25T07:36:47.371 INFO:tasks.workunit.client.0.smithi031.stderr:1024+0 records out 2024-04-25T07:36:47.371 INFO:tasks.workunit.client.0.smithi031.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.577658 s, 1.9 GB/s 2024-04-25T07:36:47.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-04-25T07:36:47.396 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:56.189 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-04-25T07:36:56.189 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:36:56.190 INFO:tasks.workunit.client.0.smithi031.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-04-25T07:36:56.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:36:57.509 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:36:57.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:36:57.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:57.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:57.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:36:57.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:57.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:57.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:57.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:57.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:57.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:57.530 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:57.530 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:57.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:36:57.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:36:57.533 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:36:57.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:36:57.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:57.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:36:57.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:36:57.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:57.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:57.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:57.572 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:36:57.570+0000 7f4ca2e147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:57.575 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:36:57.574+0000 7f4ca2e147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:57.577 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:36:57.574+0000 7f4ca2e147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:57.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:36:57.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:57.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:36:57.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:57.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:57.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:57.973 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:36:57.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:57.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:57.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:58.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:58.624 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:36:58.622+0000 7f4ca2e147c0 -1 Falling back to public interface 2024-04-25T07:36:59.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:59.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:59.472 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:36:59.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:59.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:59.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:59.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:00.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:00.894 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:37:00.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:00.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:00.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:00.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:01.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:02.374 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:37:02.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:02.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:02.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:03.401 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:03.398+0000 7f4ca2e147c0 -1 osd.0 77 log_to_monitors true 2024-04-25T07:37:03.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:03.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:03.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:03.808 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:37:03.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:03.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:04.296 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 88 up_thru 88 down_at 78 last_clean_interval [64,77) [v2:127.0.0.1:6802/2686003627,v1:127.0.0.1:6803/2686003627] [v2:127.0.0.1:6804/2686003627,v1:127.0.0.1:6805/2686003627] exists,up 937497c4-5df9-44b2-93ae-7997930c288d 2024-04-25T07:37:04.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:04.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:04.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:04.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=137312 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:04.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:04.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:04.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:04.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:04.302 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:04.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:04.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:04.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:04.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:37:04.304 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:37:04.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:37:04.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:04.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:37:04.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:37:04.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:04.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:04.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:04.328 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:04.326+0000 7f80eec1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:04.331 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:04.330+0000 7f80eec1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:04.332 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:04.330+0000 7f80eec1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:04.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:37:04.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:04.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:37:04.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:04.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:04.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:04.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:04.808 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:37:04.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:04.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:04.906 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:04.906+0000 7f80eec1c7c0 -1 Falling back to public interface 2024-04-25T07:37:05.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:06.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:06.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:06.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:06.255 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:37:06.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:06.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:06.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:07.678 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:37:07.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:07.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:08.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:08.212 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:08.209+0000 7f80eec1c7c0 -1 osd.1 80 log_to_monitors true 2024-04-25T07:37:09.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:09.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:09.100 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:37:09.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:09.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:09.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:09.659 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 81 last_clean_interval [67,80) [v2:127.0.0.1:6810/2647295988,v1:127.0.0.1:6811/2647295988] [v2:127.0.0.1:6812/2647295988,v1:127.0.0.1:6813/2647295988] exists,up cd0b508e-f02d-4fab-a50a-3024834d86ee 2024-04-25T07:37:09.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:09.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:09.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:09.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=138281 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:09.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:37:09.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:37:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:37:09.667 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:37:09.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:09.667 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:37:09.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:37:09.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:09.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:09.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:09.691 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:09.689+0000 7f3d350da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:09.748 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:09.745+0000 7f3d350da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:09.750 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:09.749+0000 7f3d350da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:37:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:37:10.098 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:37:10.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:10.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:10.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:10.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:10.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:10.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:10.539 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:10.537+0000 7f3d350da7c0 -1 Falling back to public interface 2024-04-25T07:37:10.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:11.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:11.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:11.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:11.566 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:37:11.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:11.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:13.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:13.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:13.056 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:37:13.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:13.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:13.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:13.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:14.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:14.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:14.523 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:37:14.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:14.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:14.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:14.648 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:14.645+0000 7f3d350da7c0 -1 osd.2 83 log_to_monitors true 2024-04-25T07:37:14.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:15.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:15.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:15.993 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:37:15.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:15.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:15.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:16.664 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 95 up_thru 95 down_at 84 last_clean_interval [70,83) [v2:127.0.0.1:6818/974360837,v1:127.0.0.1:6819/974360837] [v2:127.0.0.1:6820/974360837,v1:127.0.0.1:6821/974360837] exists,up c8e57ed6-531d-4749-a7ba-e3679de30c73 2024-04-25T07:37:16.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:16.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:16.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:16.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:37:16.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=139215 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:37:16.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:16.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:16.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:16.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:16.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:16.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:16.670 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:16.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:37:16.672 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:16.702 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:16.701+0000 7fcdd04b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:16.735 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:16.733+0000 7fcdd04b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:16.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:16.733+0000 7fcdd04b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:17.520 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:17.517+0000 7fcdd04b77c0 -1 Falling back to public interface 2024-04-25T07:37:17.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:18.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:18.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:18.575 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:37:18.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:18.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:18.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:19.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:20.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:20.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:20.047 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:37:20.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:20.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:20.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:20.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:21.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:21.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:21.468 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:37:21.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:21.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:21.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:21.506 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:37:21.505+0000 7fcdd04b77c0 -1 osd.3 85 log_to_monitors true 2024-04-25T07:37:21.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:22.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:22.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:22.988 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:37:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:23.419 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 99 up_thru 99 down_at 86 last_clean_interval [74,85) [v2:127.0.0.1:6826/1677652748,v1:127.0.0.1:6827/1677652748] [v2:127.0.0.1:6828/1677652748,v1:127.0.0.1:6829/1677652748] exists,up 37b02a8d-a8c0-487b-bbd9-117b387db091 2024-04-25T07:37:23.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:23.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:23.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:23.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:37:23.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=140179 2024-04-25T07:37:23.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:37:23.454 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:37:23.454 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:37:23.454 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_141128 2024-04-25T07:37:23.454 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:37:23.454 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:37:24.454 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 96 80 0.312471 0.3125 0.178838 0.180191 2024-04-25T07:37:25.454 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 212 196 0.382773 0.453125 0.160325 0.158438 2024-04-25T07:37:26.454 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 305 289 0.376256 0.363281 0.212801 0.160407 2024-04-25T07:37:27.454 INFO:tasks.workunit.client.0.smithi031.stdout: 4 16 390 374 0.365187 0.332031 0.168589 0.166436 2024-04-25T07:37:28.455 INFO:tasks.workunit.client.0.smithi031.stdout: 5 16 486 470 0.367139 0.375 0.229404 0.165395 2024-04-25T07:37:29.455 INFO:tasks.workunit.client.0.smithi031.stdout: 6 16 605 589 0.38341 0.464844 0.112292 0.161187 2024-04-25T07:37:30.455 INFO:tasks.workunit.client.0.smithi031.stdout: 7 16 701 685 0.382201 0.375 0.217997 0.160343 2024-04-25T07:37:31.455 INFO:tasks.workunit.client.0.smithi031.stdout: 8 16 803 787 0.384223 0.398438 0.154798 0.16065 2024-04-25T07:37:32.455 INFO:tasks.workunit.client.0.smithi031.stdout: 9 16 911 895 0.388401 0.421875 0.17283 0.158089 2024-04-25T07:37:33.455 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16 1047 1031 0.402678 0.53125 0.11712 0.154252 2024-04-25T07:37:34.455 INFO:tasks.workunit.client.0.smithi031.stdout: 11 16 1143 1127 0.400156 0.375 0.209474 0.154631 2024-04-25T07:37:35.455 INFO:tasks.workunit.client.0.smithi031.stdout: 12 16 1266 1250 0.406843 0.480469 0.125339 0.153141 2024-04-25T07:37:36.456 INFO:tasks.workunit.client.0.smithi031.stdout: 13 16 1335 1319 0.396278 0.269531 0.52071 0.155734 2024-04-25T07:37:37.456 INFO:tasks.workunit.client.0.smithi031.stdout: 14 16 1441 1425 0.397544 0.414062 0.124273 0.156408 2024-04-25T07:37:38.456 INFO:tasks.workunit.client.0.smithi031.stdout: 15 16 1555 1539 0.400723 0.445312 0.127496 0.154936 2024-04-25T07:37:39.456 INFO:tasks.workunit.client.0.smithi031.stdout: 16 16 1658 1642 0.400821 0.402344 0.16779 0.154945 2024-04-25T07:37:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: 17 16 1758 1742 0.400218 0.390625 0.217163 0.15525 2024-04-25T07:37:41.456 INFO:tasks.workunit.client.0.smithi031.stdout: 18 16 1878 1862 0.404021 0.46875 0.119855 0.154384 2024-04-25T07:37:42.457 INFO:tasks.workunit.client.0.smithi031.stdout: 19 16 1974 1958 0.402491 0.375 0.15852 0.154404 2024-04-25T07:37:43.457 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:37:43.459855+0000 min lat: 0.057485 max lat: 0.521345 avg lat: 0.153753 2024-04-25T07:37:43.457 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:37:43.457 INFO:tasks.workunit.client.0.smithi031.stdout: 20 16 2093 2077 0.405606 0.464844 0.133851 0.153753 2024-04-25T07:37:44.457 INFO:tasks.workunit.client.0.smithi031.stdout: 21 16 2186 2170 0.403587 0.363281 0.175137 0.153975 2024-04-25T07:37:45.457 INFO:tasks.workunit.client.0.smithi031.stdout: 22 16 2306 2290 0.406546 0.46875 0.140216 0.153335 2024-04-25T07:37:46.457 INFO:tasks.workunit.client.0.smithi031.stdout: 23 16 2403 2387 0.405342 0.378906 0.185058 0.153801 2024-04-25T07:37:47.457 INFO:tasks.workunit.client.0.smithi031.stdout: 24 16 2483 2467 0.401472 0.3125 0.227103 0.154898 2024-04-25T07:37:48.457 INFO:tasks.workunit.client.0.smithi031.stdout: 25 16 2580 2564 0.400568 0.378906 0.182332 0.155349 2024-04-25T07:37:49.458 INFO:tasks.workunit.client.0.smithi031.stdout: 26 16 2694 2678 0.402287 0.445312 0.144145 0.154893 2024-04-25T07:37:50.458 INFO:tasks.workunit.client.0.smithi031.stdout: 27 16 2785 2769 0.400552 0.355469 0.194996 0.155261 2024-04-25T07:37:51.458 INFO:tasks.workunit.client.0.smithi031.stdout: 28 16 2913 2897 0.404101 0.5 0.11712 0.154373 2024-04-25T07:37:52.458 INFO:tasks.workunit.client.0.smithi031.stdout: 29 16 3006 2990 0.402692 0.363281 0.202384 0.154294 2024-04-25T07:37:53.458 INFO:tasks.workunit.client.0.smithi031.stdout: 30 16 3129 3113 0.405283 0.480469 0.138313 0.153887 2024-04-25T07:37:54.458 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 30.1107 2024-04-25T07:37:54.458 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 3129 2024-04-25T07:37:54.458 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4096 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4096 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.405924 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0619392 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.53125 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.269531 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 103 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 15.8564 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 136 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 69 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.153841 2024-04-25T07:37:54.459 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0466694 2024-04-25T07:37:54.460 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.521345 2024-04-25T07:37:54.460 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.057485 2024-04-25T07:37:54.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-04-25T07:37:54.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:37:54.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:37:54.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:37:54.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:37:54.518 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:37:54.518 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:37:54.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:37:54.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:37:54.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:37:54.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:37:54.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:37:54.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:37:54.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:37:54.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:37:54.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:37:55.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:37:55.084 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:37:55.084 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T07:37:55.084 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T07:37:55.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:37:55.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:55.085 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:37:55.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=377957122061 2024-04-25T07:37:55.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 377957122061 2024-04-25T07:37:55.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-377957122061' 2024-04-25T07:37:55.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:55.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:37:55.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=390842023948 2024-04-25T07:37:55.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 390842023948 2024-04-25T07:37:55.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-377957122061 1-390842023948' 2024-04-25T07:37:55.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:55.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=408021893131 2024-04-25T07:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 408021893131 2024-04-25T07:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-377957122061 1-390842023948 2-408021893131' 2024-04-25T07:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:37:55.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=425201762313 2024-04-25T07:37:55.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 425201762313 2024-04-25T07:37:55.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-377957122061 1-390842023948 2-408021893131 3-425201762313' 2024-04-25T07:37:55.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:55.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:55.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-377957122061 2024-04-25T07:37:55.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:37:55.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-377957122061 2024-04-25T07:37:55.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:55.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=377957122061 2024-04-25T07:37:55.762 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 377957122061 2024-04-25T07:37:55.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 377957122061' 2024-04-25T07:37:55.763 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:37:56.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 377957122061 -lt 377957122061 2024-04-25T07:37:56.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:56.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-390842023948 2024-04-25T07:37:56.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:56.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:37:56.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-390842023948 2024-04-25T07:37:56.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:56.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=390842023948 2024-04-25T07:37:56.260 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 390842023948 2024-04-25T07:37:56.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 390842023948' 2024-04-25T07:37:56.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:37:56.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 390842023948 -lt 390842023948 2024-04-25T07:37:56.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:56.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-408021893131 2024-04-25T07:37:56.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:37:56.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-408021893131 2024-04-25T07:37:56.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:56.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=408021893131 2024-04-25T07:37:56.722 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 408021893131 2024-04-25T07:37:56.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 408021893131' 2024-04-25T07:37:56.722 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:37:57.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 408021893131 -lt 408021893131 2024-04-25T07:37:57.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:57.234 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-425201762313 2024-04-25T07:37:57.234 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:57.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:37:57.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:57.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-425201762313 2024-04-25T07:37:57.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=425201762313 2024-04-25T07:37:57.238 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 425201762313 2024-04-25T07:37:57.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 425201762313' 2024-04-25T07:37:57.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:37:57.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 425201762313 -lt 425201762313 2024-04-25T07:37:57.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:37:57.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:37:57.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:37:58.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:37:58.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:37:58.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:37:58.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:37:58.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:37:58.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:37:58.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:37:58.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:37:58.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:37:58.708 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:37:58.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:37:58.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:37:59.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:37:59.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:37:59.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:37:59.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:37:59.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:59.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:37:59.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:37:59.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:37:59.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:37:59.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:37:59.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:37:59.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:37:59.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:37:59.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:37:59.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:37:59.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:37:59.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:37:59.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:37:59.584 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:37:59.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:37:59.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:37:59.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:37:59.585 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:37:59.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:37:59.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:37:59.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:38:00.085 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:38:00.085 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:00.085 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:00.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:38:00.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:38:00.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:38:00.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:38:00.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:38:00.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:38:00.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:38:00.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:38:00.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:38:00.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:38:00.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:38:00.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:38:00.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:38:00.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:38:00.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:38:00.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:38:00.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:38:00.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:38:00.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:38:00.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:38:00.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:00.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:00.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:38:00.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:38:00.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:38:00.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:38:00.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:38:00.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:00.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:00.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94549 2024-04-25T07:38:00.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:38:00.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:38:00.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:38:00.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:38:00.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-04-25T07:38:00.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:00.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-04-25T07:38:00.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-04-25T07:38:00.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-04-25T07:38:00.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:38:00.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-04-25T07:38:00.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-04-25T07:38:00.105 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none ' 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:38:00.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:38:00.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:00.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:38:00.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:00.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:00.592 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:00.592 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:00.592 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:00.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:38:00.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:38:00.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:38:00.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:38:00.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:38:00.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:38:00.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:38:00.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:38:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:38:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:38:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:38:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:38:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94549/ceph-mon.a.asok 2024-04-25T07:38:00.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:38:00.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94549/ceph-mon.a.asok config get fsid 2024-04-25T07:38:00.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:38:00.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:38:00.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:38:00.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:38:00.925 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:38:00.925 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:38:00.925 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94549/ceph-mon.a.asok 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:38:00.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94549/ceph-mon.a.asok config get mon_host 2024-04-25T07:38:01.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:38:01.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:01.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:38:01.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:38:01.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:38:01.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:38:01.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:38:01.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:38:01.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:01.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:01.982 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:01.982 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:01.982 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:01.982 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:38:01.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:38:01.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:38:02.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:38:02.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:02.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:38:02.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:38:02.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:38:02.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:38:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:38:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:38:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:38:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:38:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:38:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=614c7252-57c3-460a-8a0d-e7076f5a0131 2024-04-25T07:38:02.051 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 614c7252-57c3-460a-8a0d-e7076f5a0131 2024-04-25T07:38:02.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 614c7252-57c3-460a-8a0d-e7076f5a0131' 2024-04-25T07:38:02.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:38:02.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaCCpmWggXBBAAkT4rtsI/PwVkeTGf+crWdg== 2024-04-25T07:38:02.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaCCpmWggXBBAAkT4rtsI/PwVkeTGf+crWdg=="}' 2024-04-25T07:38:02.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 614c7252-57c3-460a-8a0d-e7076f5a0131 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:38:02.372 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:38:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:38:02.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaCCpmWggXBBAAkT4rtsI/PwVkeTGf+crWdg== --osd-uuid 614c7252-57c3-460a-8a0d-e7076f5a0131 2024-04-25T07:38:02.422 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.421+0000 7fe30aeb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:02.433 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.429+0000 7fe30aeb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:02.434 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.432+0000 7fe30aeb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:02.434 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.432+0000 7fe30aeb87c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:38:02.435 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.432+0000 7fe30aeb87c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:38:02.435 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.432+0000 7fe30aeb87c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:38:02.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:02.468+0000 7fe30aeb87c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:38:10.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:38:10.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:38:10.778 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:38:10.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:38:10.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:38:11.364 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:38:11.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:38:11.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:11.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:38:11.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:38:11.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:38:11.394 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:11.392+0000 7efec29187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:11.429 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:11.428+0000 7efec29187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:11.431 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:11.428+0000 7efec29187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:11.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:38:11.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:11.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:38:11.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:11.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:11.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:11.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:11.809 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:38:11.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:11.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:12.231 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:12.228+0000 7efec29187c0 -1 Falling back to public interface 2024-04-25T07:38:12.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:13.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:13.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:13.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:13.268 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:38:13.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:13.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:13.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:14.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:14.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:14.706 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:38:14.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:14.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:14.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:15.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:16.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:16.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:16.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:16.171 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:38:16.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:16.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:16.308 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:38:16.304+0000 7efec29187c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:38:16.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:17.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:17.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:17.657 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:38:17.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:17.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:17.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:19.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:20.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:20.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:20.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:38:20.075 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:38:20.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:20.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:20.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:21.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:21.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:21.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:38:21.671 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:38:21.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:21.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:22.111 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1919468885,v1:127.0.0.1:6803/1919468885] [v2:127.0.0.1:6804/1919468885,v1:127.0.0.1:6805/1919468885] exists,up 614c7252-57c3-460a-8a0d-e7076f5a0131 2024-04-25T07:38:22.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:22.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:22.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:22.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:38:22.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=142997 2024-04-25T07:38:22.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-04-25T07:38:27.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-04-25T07:38:27.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:38:27.680 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T07:38:27.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:38:28.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-04-25T07:38:28.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-04-25T07:38:28.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-04-25T07:38:28.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-04-25T07:38:28.783 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:38:28.785 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-04-25T07:38:28.785 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_144280 2024-04-25T07:38:28.823 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:38:28.823 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:38:29.823 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 17 1 3.99945 4 0.31204 0.31204 2024-04-25T07:38:30.824 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 33 17 33.9947 64 1.03612 1.13229 2024-04-25T07:38:31.824 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 49 33 43.9933 64 0.960456 1.03977 2024-04-25T07:38:32.824 INFO:tasks.workunit.client.0.smithi031.stdout: 4 16 65 49 48.9925 64 1.03549 1.06419 2024-04-25T07:38:33.824 INFO:tasks.workunit.client.0.smithi031.stdout: 5 16 81 65 51.992 64 0.893494 1.01859 2024-04-25T07:38:34.824 INFO:tasks.workunit.client.0.smithi031.stdout: 6 16 97 81 53.9916 64 1.43675 1.09003 2024-04-25T07:38:35.824 INFO:tasks.workunit.client.0.smithi031.stdout: 7 16 113 97 55.42 64 1.27805 1.13067 2024-04-25T07:38:36.824 INFO:tasks.workunit.client.0.smithi031.stdout: 8 16 129 113 56.4915 64 0.851712 1.09109 2024-04-25T07:38:37.824 INFO:tasks.workunit.client.0.smithi031.stdout: 9 16 144 128 56.8803 60 1.27105 1.10866 2024-04-25T07:38:38.825 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16 161 145 57.9913 68 0.834829 1.07885 2024-04-25T07:38:39.825 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 10.9861 2024-04-25T07:38:39.825 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 161 2024-04-25T07:38:39.825 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4194304 2024-04-25T07:38:39.825 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4194304 2024-04-25T07:38:39.825 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 58.6193 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 19.0671 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 68 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 4 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 14 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 4.76678 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 17 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 1 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 1.08482 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.202608 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 1.43675 2024-04-25T07:38:39.826 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.31204 2024-04-25T07:38:39.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-04-25T07:38:49.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:38:49.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-04-25T07:38:50.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1060110336 2024-04-25T07:38:50.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:38:50.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-04-25T07:38:50.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=73924608 2024-04-25T07:38:50.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-04-25T07:38:50.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 73924608 -eq 0 2024-04-25T07:38:50.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-04-25T07:38:50.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 73924608 -gt 0 2024-04-25T07:38:50.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 142997 2024-04-25T07:38:50.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-04-25T07:38:51.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 142997 2024-04-25T07:38:51.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (142997) - No such process 2024-04-25T07:38:51.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-04-25T07:38:51.688 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:38:51.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-04-25T07:38:51.735 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:01.270 INFO:tasks.workunit.client.0.smithi031.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:39:01.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-04-25T07:39:01.330 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:03.115 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0xffffe000 : using 0x43b97000(1.1 GiB) 2024-04-25T07:39:03.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-04-25T07:39:05.209 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:39:05.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:39:05.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:39:05.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:39:05.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:39:05.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:39:05.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:39:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:39:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:39:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:39:05.260 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:39:05.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:39:05.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:39:05.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:39:05.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:39:05.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:39:05.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:39:05.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:39:05.299 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:05.299+0000 7f479dabf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:05.303+0000 7f479dabf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:05.304 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:05.303+0000 7f479dabf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:39:05.690 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:39:05.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:05.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:05.847 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:05.847+0000 7f479dabf7c0 -1 Falling back to public interface 2024-04-25T07:39:06.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:07.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:07.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:07.122 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:39:07.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:39:07.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:07.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:07.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:08.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:08.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:08.613 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:39:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:39:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:08.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:09.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:10.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:10.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:10.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:39:10.155 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:39:10.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:10.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:10.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:11.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:11.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:11.602 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:39:11.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:39:11.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:11.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:12.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:13.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:13.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:13.048 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:39:13.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:39:13.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:13.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:13.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:14.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:14.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:14.489 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:39:14.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:39:14.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:14.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:14.648 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:14.647+0000 7f479dabf7c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:39:14.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:15.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:15.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:15.949 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:39:15.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:39:15.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:15.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:16.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:17.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:17.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:17.411 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:39:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:39:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:17.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:18.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:18.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:18.860 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:39:18.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:39:18.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:18.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:19.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:20.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:20.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:20.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:39:20.476 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:39:20.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:20.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:20.976 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/3896179483,v1:127.0.0.1:6803/3896179483] [v2:127.0.0.1:6804/3896179483,v1:127.0.0.1:6805/3896179483] exists,up 614c7252-57c3-460a-8a0d-e7076f5a0131 2024-04-25T07:39:20.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:39:20.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:39:20.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:39:20.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:39:20.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=145505 2024-04-25T07:39:20.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-04-25T07:39:20.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:39:20.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:39:20.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:39:20.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:39:20.979 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:39:20.979 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:39:20.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:39:20.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:39:20.980 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:39:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:39:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:39:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:39:21.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:39:21.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:39:21.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:39:21.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:39:21.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:39:21.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:39:21.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:39:21.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T07:39:21.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T07:39:21.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T07:39:21.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:39:21.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T07:39:21.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:39:21.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:39:21.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T07:39:21.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:39:21.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T07:39:21.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T07:39:21.787 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 55834574852 2024-04-25T07:39:21.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:39:22.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 2 -lt 55834574852 2024-04-25T07:39:22.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:39:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:39:23.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:39:23.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T07:39:23.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:39:23.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:39:23.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:39:24.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:39:24.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:39:24.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:39:24.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:39:24.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:39:24.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:39:24.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:39:24.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:39:24.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:39:24.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:39:24.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:39:24.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:39:25.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:39:25.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:39:25.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:39:25.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:39:25.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:39:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:39:25.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:39:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:39:25.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:39:25.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:39:25.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:39:25.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:39:25.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:25.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:39:25.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:39:25.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:25.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:39:25.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:39:25.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:39:25.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:39:25.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:25.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:25.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:39:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:39:25.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:39:25.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:39:25.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:39:25.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:39:25.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:39:25.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:39:25.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:39:25.470 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:39:25.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:25.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:39:25.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:39:25.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:25.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:39:25.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:39:25.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:39:25.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:39:25.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:25.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:25.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:39:25.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:39:25.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:39:25.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:39:25.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:39:25.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:25.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:25.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94549 2024-04-25T07:39:25.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:39:25.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:39:25.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:39:25.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:39:25.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-04-25T07:39:25.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-04-25T07:39:25.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=1589155c-0cd1-442b-a2c9-ed9f8979e76d --auth-supported=none ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:39:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:39:25.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:25.975 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:39:25.975 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:25.975 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:25.976 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:25.976 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:25.976 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:25.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:39:25.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:39:26.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:39:26.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:39:26.204 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:39:26.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:39:26.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:39:26.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:39:26.206 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:39:26.206 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:39:26.206 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:39:26.206 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:39:26.206 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:26.207 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:26.207 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94549/ceph-mon.a.asok 2024-04-25T07:39:26.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:39:26.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94549/ceph-mon.a.asok config get fsid 2024-04-25T07:39:26.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:39:26.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:39:26.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:39:26.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:26.326 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:26.327 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94549/ceph-mon.a.asok 2024-04-25T07:39:26.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:39:26.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94549/ceph-mon.a.asok config get mon_host 2024-04-25T07:39:26.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:39:26.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:26.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:39:26.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:39:26.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:39:26.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:39:26.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:39:27.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:39:27.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:27.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:27.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:27.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:27.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:27.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:39:27.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:39:27.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:39:27.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:39:27.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:27.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:39:27.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=1589155c-0cd1-442b-a2c9-ed9f8979e76d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:27.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:39:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:39:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:39:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:39:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:39:27.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:39:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0645650a-fc31-4a44-9b04-4b26c38a07d9 2024-04-25T07:39:27.448 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 0645650a-fc31-4a44-9b04-4b26c38a07d9 2024-04-25T07:39:27.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0645650a-fc31-4a44-9b04-4b26c38a07d9' 2024-04-25T07:39:27.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:39:27.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvCCpmOppVHBAAjD0KBiKxyzpC4I8ZHz9cow== 2024-04-25T07:39:27.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvCCpmOppVHBAAjD0KBiKxyzpC4I8ZHz9cow=="}' 2024-04-25T07:39:27.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0645650a-fc31-4a44-9b04-4b26c38a07d9 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:39:27.814 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:39:27.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:39:27.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=1589155c-0cd1-442b-a2c9-ed9f8979e76d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvCCpmOppVHBAAjD0KBiKxyzpC4I8ZHz9cow== --osd-uuid 0645650a-fc31-4a44-9b04-4b26c38a07d9 2024-04-25T07:39:27.880 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.879+0000 7f1d79ef07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:27.891 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.891+0000 7f1d79ef07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:27.892 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.891+0000 7f1d79ef07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:27.892 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.891+0000 7f1d79ef07c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:39:27.892 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.891+0000 7f1d79ef07c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:39:27.893 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.891+0000 7f1d79ef07c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:39:27.931 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:27.931+0000 7f1d79ef07c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:39:36.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:39:36.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:39:36.586 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:39:36.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:39:36.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:39:37.176 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:39:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:39:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=1589155c-0cd1-442b-a2c9-ed9f8979e76d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:39:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:39:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:39:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:39:37.221 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:37.219+0000 7f65eaab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:37.254 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:37.255+0000 7f65eaab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:37.256 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:37.255+0000 7f65eaab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:37.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:39:37.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:39:37.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:39:37.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:39:37.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:39:37.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:37.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:39:37.648 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:39:37.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:37.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:38.055+0000 7f65eaab47c0 -1 Falling back to public interface 2024-04-25T07:39:38.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:39.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:39.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:39.079 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:39:39.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:39:39.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:39.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:40.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:40.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:40.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:39:40.517 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:39:40.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:40.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:40.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:41.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:41.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:39:41.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:39:41.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:41.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:42.067 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:39:42.067+0000 7f65eaab47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:39:42.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:43.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:43.421 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:39:43.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:43.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:39:43.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:43.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:44.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:45.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:45.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:45.693 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:39:45.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:39:45.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:45.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:46.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:47.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:47.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:47.521 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:39:47.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:39:47.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:47.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:48.000 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3031279663,v1:127.0.0.1:6803/3031279663] [v2:127.0.0.1:6804/3031279663,v1:127.0.0.1:6805/3031279663] exists,up 0645650a-fc31-4a44-9b04-4b26c38a07d9 2024-04-25T07:39:48.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:39:48.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:39:48.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:39:48.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:39:48.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=148297 2024-04-25T07:39:48.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-04-25T07:39:53.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-04-25T07:39:53.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:39:53.586 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T07:39:53.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:39:54.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:39:54.656 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:39:54.656 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:39:54.656 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_149323 2024-04-25T07:39:54.656 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:39:54.656 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:39:55.656 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 129 113 0.441352 0.441406 0.142191 0.125323 2024-04-25T07:39:56.656 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 241 225 0.439395 0.4375 0.167198 0.133786 2024-04-25T07:39:57.656 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 369 353 0.459572 0.5 0.108753 0.134053 2024-04-25T07:39:58.657 INFO:tasks.workunit.client.0.smithi031.stdout: 4 16 496 480 0.468682 0.496094 0.14202 0.130921 2024-04-25T07:39:59.657 INFO:tasks.workunit.client.0.smithi031.stdout: 5 16 625 609 0.475713 0.503906 0.13385 0.128089 2024-04-25T07:40:00.657 INFO:tasks.workunit.client.0.smithi031.stdout: 6 16 721 705 0.458918 0.375 0.417725 0.132713 2024-04-25T07:40:01.657 INFO:tasks.workunit.client.0.smithi031.stdout: 7 16 848 832 0.464217 0.496094 0.142165 0.133952 2024-04-25T07:40:02.657 INFO:tasks.workunit.client.0.smithi031.stdout: 8 16 961 945 0.461357 0.441406 0.159093 0.133949 2024-04-25T07:40:03.657 INFO:tasks.workunit.client.0.smithi031.stdout: 9 16 1089 1073 0.465642 0.5 0.100411 0.132918 2024-04-25T07:40:04.657 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16 1201 1185 0.462823 0.4375 0.159007 0.132479 2024-04-25T07:40:05.658 INFO:tasks.workunit.client.0.smithi031.stdout: 11 16 1329 1313 0.466198 0.5 0.117026 0.133394 2024-04-25T07:40:06.658 INFO:tasks.workunit.client.0.smithi031.stdout: 12 16 1456 1440 0.468684 0.496094 0.184561 0.133054 2024-04-25T07:40:07.658 INFO:tasks.workunit.client.0.smithi031.stdout: 13 16 1584 1568 0.471088 0.5 0.11711 0.132287 2024-04-25T07:40:08.658 INFO:tasks.workunit.client.0.smithi031.stdout: 14 16 1696 1680 0.468684 0.4375 0.167628 0.132455 2024-04-25T07:40:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: 15 16 1824 1808 0.470767 0.5 0.118536 0.132418 2024-04-25T07:40:10.658 INFO:tasks.workunit.client.0.smithi031.stdout: 16 16 1937 1921 0.468927 0.441406 0.167338 0.132509 2024-04-25T07:40:11.658 INFO:tasks.workunit.client.0.smithi031.stdout: 17 16 2065 2049 0.470751 0.5 0.1339 0.132005 2024-04-25T07:40:12.658 INFO:tasks.workunit.client.0.smithi031.stdout: 18 16 2177 2161 0.468899 0.4375 0.175444 0.132173 2024-04-25T07:40:13.659 INFO:tasks.workunit.client.0.smithi031.stdout: 19 16 2304 2288 0.470327 0.496094 0.118641 0.132601 2024-04-25T07:40:14.659 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:40:14.662037+0000 min lat: 0.0903704 max lat: 0.417725 avg lat: 0.133166 2024-04-25T07:40:14.659 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:40:14.659 INFO:tasks.workunit.client.0.smithi031.stdout: 20 16 2416 2400 0.468682 0.4375 0.177643 0.133166 2024-04-25T07:40:15.659 INFO:tasks.workunit.client.0.smithi031.stdout: 21 16 2528 2512 0.467193 0.4375 0.118299 0.133258 2024-04-25T07:40:16.659 INFO:tasks.workunit.client.0.smithi031.stdout: 22 16 2625 2609 0.463177 0.378906 0.200501 0.133757 2024-04-25T07:40:17.659 INFO:tasks.workunit.client.0.smithi031.stdout: 23 16 2753 2737 0.464775 0.5 0.117074 0.133838 2024-04-25T07:40:18.660 INFO:tasks.workunit.client.0.smithi031.stdout: 24 16 2881 2865 0.46624 0.5 0.160499 0.133659 2024-04-25T07:40:19.660 INFO:tasks.workunit.client.0.smithi031.stdout: 25 16 3008 2992 0.467431 0.496094 0.151031 0.133579 2024-04-25T07:40:20.660 INFO:tasks.workunit.client.0.smithi031.stdout: 26 16 3120 3104 0.466277 0.4375 0.184591 0.133929 2024-04-25T07:40:21.660 INFO:tasks.workunit.client.0.smithi031.stdout: 27 16 3248 3232 0.467523 0.5 0.125494 0.133519 2024-04-25T07:40:22.660 INFO:tasks.workunit.client.0.smithi031.stdout: 28 16 3360 3344 0.466448 0.4375 0.150748 0.133846 2024-04-25T07:40:23.660 INFO:tasks.workunit.client.0.smithi031.stdout: 29 16 3489 3473 0.467737 0.503906 0.108688 0.133309 2024-04-25T07:40:24.660 INFO:tasks.workunit.client.0.smithi031.stdout: 30 16 3601 3585 0.466727 0.4375 0.133897 0.133222 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 30.0809 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 3601 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4096 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4096 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.46762 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0384071 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.503906 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.375 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 119 2024-04-25T07:40:25.661 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 9.83221 2024-04-25T07:40:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 129 2024-04-25T07:40:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 96 2024-04-25T07:40:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.133443 2024-04-25T07:40:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0310474 2024-04-25T07:40:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.417725 2024-04-25T07:40:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.0903704 2024-04-25T07:40:25.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-04-25T07:40:30.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:40:30.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:40:30.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-04-25T07:40:30.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:40:30.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:40:30.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-04-25T07:40:30.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:40:30.905 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:40:30.905 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:40:31.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4280123392 2024-04-25T07:40:31.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 148297 2024-04-25T07:40:31.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-04-25T07:40:32.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 148297 2024-04-25T07:40:32.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (148297) - No such process 2024-04-25T07:40:32.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:40:32.531 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:40:32.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-04-25T07:40:32.583 INFO:tasks.workunit.client.0.smithi031.stdout:allocmap bluestore.allocmap 2024-04-25T07:40:35.365 INFO:tasks.workunit.client.0.smithi031.stdout:allocmap success 2024-04-25T07:40:35.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:40:38.201 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:40:38.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-04-25T07:40:38.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:40:38.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:40:38.265 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:40.351 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:40:40.351 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0x100000000 : using 0xe28000(14 MiB) 2024-04-25T07:40:40.351 INFO:tasks.workunit.client.0.smithi031.stdout:Expanding DB/WAL... 2024-04-25T07:40:40.352 INFO:tasks.workunit.client.0.smithi031.stdout:1 : expanding to 0x1073741824 2024-04-25T07:40:40.384 INFO:tasks.workunit.client.0.smithi031.stdout:1 : size label updated to 1073741824 2024-04-25T07:40:41.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:40:44.129 INFO:tasks.workunit.client.0.smithi031.stdout:fsck success 2024-04-25T07:40:44.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:40:44.186 INFO:tasks.workunit.client.0.smithi031.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:40:46.969 INFO:tasks.workunit.client.0.smithi031.stdout:qfsck success 2024-04-25T07:40:46.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:40:47.020 INFO:tasks.workunit.client.0.smithi031.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:49.807 INFO:tasks.workunit.client.0.smithi031.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:40:49.807 INFO:tasks.workunit.client.0.smithi031.stdout:2 : device size 0x100000000 : using 0xe28000(14 MiB) 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=1589155c-0cd1-442b-a2c9-ed9f8979e76d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:40:49.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:40:49.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:40:49.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:40:49.823 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:40:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=65f6c278-055a-42a3-8843-dbfb81910280 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=d5099b79-9e01-4af5-ab80-95eb3827d5f3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=1589155c-0cd1-442b-a2c9-ed9f8979e76d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94549/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:49.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:40:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:40:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:49.866 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:40:49.866+0000 7f08b083c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:49.868 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:40:49.866+0000 7f08b083c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:49.869 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:40:49.870+0000 7f08b083c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:50.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:40:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:50.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:50.672 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:40:50.670+0000 7f08b083c7c0 -1 Falling back to public interface 2024-04-25T07:40:50.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:51.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:51.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:51.745 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:40:51.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:51.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:51.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:53.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:53.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:53.186 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:40:53.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:53.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:53.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:53.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:54.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:54.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:54.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:54.633 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:40:54.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:54.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:55.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:55.640 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:40:55.638+0000 7f08b083c7c0 -1 osd.0 9 log_to_monitors true 2024-04-25T07:40:56.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:56.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:56.077 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:40:56.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:56.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:56.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:56.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:57.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:57.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:57.889 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:40:57.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:40:57.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:57.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:58.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:59.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:59.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:59.335 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:40:59.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:40:59.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:59.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:59.853 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 12 up_thru 7 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/3630604167,v1:127.0.0.1:6803/3630604167] [v2:127.0.0.1:6804/3630604167,v1:127.0.0.1:6805/3630604167] exists,up 0645650a-fc31-4a44-9b04-4b26c38a07d9 2024-04-25T07:40:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:59.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:59.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:40:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=151676 2024-04-25T07:40:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-04-25T07:40:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:40:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:40:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:40:59.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:40:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:40:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:40:59.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:40:59.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:40:59.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:40:59.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:40:59.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:40:59.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:40:59.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:40:59.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:40:59.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:41:00.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:41:00.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:41:00.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:41:00.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:41:00.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607555 2024-04-25T07:41:00.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607555 2024-04-25T07:41:00.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-51539607555' 2024-04-25T07:41:00.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:41:00.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-51539607555 2024-04-25T07:41:00.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:41:00.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:41:00.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-51539607555 2024-04-25T07:41:00.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:41:00.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607555 2024-04-25T07:41:00.676 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 51539607555 2024-04-25T07:41:00.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 51539607555' 2024-04-25T07:41:00.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:41:01.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 51539607555 2024-04-25T07:41:01.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:41:02.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:41:02.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:41:02.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607556 -lt 51539607555 2024-04-25T07:41:02.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:41:02.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:41:02.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:41:03.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:41:03.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:41:03.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:41:03.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:41:03.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:41:03.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:41:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:41:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:41:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:41:04.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:41:04.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:41:04.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-04-25T07:41:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:41:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-04-25T07:41:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:41:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:41:04.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:41:04.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:41:04.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4280078336 2024-04-25T07:41:04.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-04-25T07:41:04.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-04-25T07:41:04.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-04-25T07:41:04.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-45056 2024-04-25T07:41:04.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -45056)' 2024-04-25T07:41:04.622 INFO:tasks.workunit.client.0.smithi031.stdout:45056 2024-04-25T07:41:04.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 45056 2024-04-25T07:41:04.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 45056 -gt 131072 ']' 2024-04-25T07:41:04.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 151676 2024-04-25T07:41:04.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-04-25T07:41:05.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 151676 2024-04-25T07:41:05.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (151676) - No such process 2024-04-25T07:41:05.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:41:06.073 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T07:41:06.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:41:06.126 INFO:tasks.workunit.client.0.smithi031.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:41:08.157 INFO:tasks.workunit.client.0.smithi031.stdout:qfsck success 2024-04-25T07:41:08.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:41:08.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:08.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:41:08.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:41:08.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:08.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:08.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:08.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:08.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:08.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:08.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:08.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:08.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:08.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:08.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:08.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:08.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:08.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:08.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:08.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:08.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:41:08.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:41:08.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:08.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:08.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:41:08.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:41:08.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:08.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:08.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:41:08.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bluefs-volume-ops 0 2024-04-25T07:41:08.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:08.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:41:08.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:41:08.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:08.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:08.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:08.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:08.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:08.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:08.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:08.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:08.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:08.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:08.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:08.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:08.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:08.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:08.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:08.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:08.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:08.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:41:08.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:41:08.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:08.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:08.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94549 2024-04-25T07:41:08.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94549 2024-04-25T07:41:08.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:08.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:08.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:41:08.396 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:41:08.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:41:08.449 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-04-25T07:41:08.450 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-04-25T07:41:08.505 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:41:08.510 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:41:08.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:41:08.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-config 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-04-25T07:41:08.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-04-25T07:41:08.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none ' 2024-04-25T07:41:08.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T07:41:08.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-04-25T07:41:08.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:41:08.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-04-25T07:41:08.515 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_config_track 2024-04-25T07:41:08.515 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_default_adjustment' 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:08.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:08.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:08.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:08.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:08.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:08.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:08.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:08.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:08.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:08.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:08.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:08.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:08.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:41:08.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:41:08.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:08.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:08.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:08.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:41:08.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:08.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:08.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:41:08.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:41:08.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:08.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:08.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153397 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-04-25T07:41:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:41:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:41:09.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:41:09.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:09.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:09.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:09.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:09.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:09.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:09.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:41:09.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:41:09.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:41:09.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:41:09.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:41:09.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:41:09.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:41:09.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-mon.a.asok 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:41:09.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153397/ceph-mon.a.asok config get fsid 2024-04-25T07:41:09.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:41:09.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:41:09.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:41:09.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:41:09.346 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:41:09.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:41:09.346 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:41:09.346 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:41:09.347 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:09.347 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:09.347 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:09.347 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-mon.a.asok 2024-04-25T07:41:09.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:41:09.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153397/ceph-mon.a.asok config get mon_host 2024-04-25T07:41:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-04-25T07:41:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:41:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:41:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:41:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:41:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:41:09.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:41:10.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:41:10.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:10.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:10.381 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:10.381 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:10.381 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:10.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:10.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:41:10.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:41:10.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:10.420 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:41:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:41:10.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:41:10.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-04-25T07:41:10.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:41:10.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:41:10.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6082a975-6d23-4f25-bfe7-d7dc3292f783 2024-04-25T07:41:10.422 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 6082a975-6d23-4f25-bfe7-d7dc3292f783 2024-04-25T07:41:10.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6082a975-6d23-4f25-bfe7-d7dc3292f783' 2024-04-25T07:41:10.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:41:10.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWCSpmUxYmGhAAz+0+80Z+HkmFwHQqAvSUCg== 2024-04-25T07:41:10.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWCSpmUxYmGhAAz+0+80Z+HkmFwHQqAvSUCg=="}' 2024-04-25T07:41:10.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6082a975-6d23-4f25-bfe7-d7dc3292f783 -i td/osd-config/0/new.json 2024-04-25T07:41:10.761 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:41:10.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:41:10.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQAWCSpmUxYmGhAAz+0+80Z+HkmFwHQqAvSUCg== --osd-uuid 6082a975-6d23-4f25-bfe7-d7dc3292f783 2024-04-25T07:41:10.818 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.813+0000 7fd7bc4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:10.839 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.837+0000 7fd7bc4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.841+0000 7fd7bc4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.841+0000 7fd7bc4cc7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:41:10.842 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.841+0000 7fd7bc4cc7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:41:10.842 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.841+0000 7fd7bc4cc7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:41:10.878 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:10.877+0000 7fd7bc4cc7c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:41:17.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:41:17.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:41:17.540 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:41:17.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:41:17.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:41:18.119 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:41:18.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:41:18.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:41:18.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:41:18.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:41:18.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:41:18.153 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:18.149+0000 7f4b3f8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:18.178 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:18.177+0000 7f4b3f8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:18.179 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:18.177+0000 7f4b3f8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:18.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:41:18.581 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:41:18.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:18.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:18.785+0000 7f4b3f8317c0 -1 Falling back to public interface 2024-04-25T07:41:19.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:20.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:20.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:20.069 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:41:20.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:41:20.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:20.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:20.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:21.469 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:21.465+0000 7f4b3f8317c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:41:21.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:21.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:21.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:41:21.485 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:41:21.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:21.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:22.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:23.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:23.026 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:41:23.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:23.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:41:23.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:23.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:23.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:24.483 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:41:24.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:41:24.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:24.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:24.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:25.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:25.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:25.981 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:41:25.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:41:25.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:25.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:27.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:27.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:27.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:41:27.490 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:41:27.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:27.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:27.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:28.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:28.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:28.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:41:28.937 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:41:28.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:28.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:30.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:30.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:30.427 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:41:30.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:41:30.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:30.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:30.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:31.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:31.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:31.873 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:41:31.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:41:31.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:31.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:32.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:33.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:33.499 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:41:33.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:33.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:41:33.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:33.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:33.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:34.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:34.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:34.933 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:41:34.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:41:34.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:34.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:35.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:36.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:36.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:36.854 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:41:36.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:41:36.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:36.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:37.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:38.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:38.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:38.623 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:41:38.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:41:38.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:38.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:39.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:40.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:40.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:40.566 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:41:40.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:41:40.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:40.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:41.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:42.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:42.509 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:41:42.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:42.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:41:42.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:42.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:43.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:44.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:44.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:44.362 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:41:44.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:41:44.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:44.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:45.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:46.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:46.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:46.076 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:41:46.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:41:46.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:46.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:46.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:47.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:47.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:47.541 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:41:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:41:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:47.975 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1927543422,v1:127.0.0.1:6803/1927543422] [v2:127.0.0.1:6804/1927543422,v1:127.0.0.1:6805/1927543422] exists,up 6082a975-6d23-4f25-bfe7-d7dc3292f783 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:47.977 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:47.978 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:47.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-osd.0.asok 2024-04-25T07:41:47.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-04-25T07:41:47.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.153397/ceph-osd.0.asok log flush 2024-04-25T07:41:48.081 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:41:48.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:41:48.096 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:41:21.841+0000 7f4b3f8317c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:41:48.096 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:41:24.469+0000 7f4b2f7d3640 10 log_client will send 2024-04-25T07:41:21.846907+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:41:48.096 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:41:24.701+0000 7f4b307d5640 10 log_client logged 2024-04-25T07:41:21.846907+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:41:48.096 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:41:46.541+0000 7f4b3b7eb640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:41:48.096 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:41:47.469+0000 7f4b2f7d3640 10 log_client will send 2024-04-25T07:41:46.545575+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:41:48.096 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:41:47.689+0000 7f4b307d5640 10 log_client logged 2024-04-25T07:41:46.545575+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:41:48.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:41:48.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:41:48.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:41:48.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:41:48.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:48.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:48.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:48.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:48.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:48.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:48.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:48.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:48.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:48.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:48.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:48.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:48.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:48.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:48.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:48.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:48.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:48.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:41:48.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:41:48.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:48.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:48.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:48.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:41:48.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:48.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:48.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:41:48.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:48.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:48.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:48.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:48.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:48.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:48.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:48.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:48.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:48.432 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:48.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:41:48.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:41:48.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:48.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:48.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:48.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:41:48.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:48.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:48.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:41:48.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:41:48.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:48.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:48.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153397 2024-04-25T07:41:48.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:41:48.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:41:48.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:41:48.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:41:48.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:48.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:48.944 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:48.944 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:48.944 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:48.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:48.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:41:49.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:41:49.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:41:49.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:41:49.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:41:49.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:49.166 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:49.167 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-mon.a.asok 2024-04-25T07:41:49.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:41:49.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153397/ceph-mon.a.asok config get fsid 2024-04-25T07:41:49.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:41:49.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:41:49.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:41:49.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:41:49.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:41:49.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:41:49.285 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:41:49.285 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:41:49.285 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:49.285 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:49.286 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:49.286 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-mon.a.asok 2024-04-25T07:41:49.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:41:49.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153397/ceph-mon.a.asok config get mon_host 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:41:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:41:50.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:41:50.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:50.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:50.298 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:50.298 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:50.298 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:50.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:50.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:41:50.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:41:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-04-25T07:41:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:41:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:41:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:41:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:41:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:41:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:41:50.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:41:50.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:41:50.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d66684e-2505-4094-84a4-6c02ac604ef5 2024-04-25T07:41:50.336 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 0d66684e-2505-4094-84a4-6c02ac604ef5 2024-04-25T07:41:50.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0d66684e-2505-4094-84a4-6c02ac604ef5' 2024-04-25T07:41:50.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:41:50.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+CSpmsq8KFRAArpwqrd8V8NJ0UrxP67h8PA== 2024-04-25T07:41:50.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+CSpmsq8KFRAArpwqrd8V8NJ0UrxP67h8PA=="}' 2024-04-25T07:41:50.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d66684e-2505-4094-84a4-6c02ac604ef5 -i td/osd-config/0/new.json 2024-04-25T07:41:50.633 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:41:50.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:41:50.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+CSpmsq8KFRAArpwqrd8V8NJ0UrxP67h8PA== --osd-uuid 0d66684e-2505-4094-84a4-6c02ac604ef5 2024-04-25T07:41:50.675 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.673+0000 7fe1094657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:50.702 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.701+0000 7fe1094657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:50.703 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.701+0000 7fe1094657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:50.703 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.701+0000 7fe1094657c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:41:50.704 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.701+0000 7fe1094657c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:41:50.704 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.701+0000 7fe1094657c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:41:50.742 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:50.741+0000 7fe1094657c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:41:57.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:41:57.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:41:57.309 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:41:57.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:41:57.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:41:57.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:41:57.901 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:41:57.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:41:57.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=35e32402-b45f-4e19-bdda-429a245dd2a6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153397/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:41:57.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:41:57.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:41:57.931 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:57.929+0000 7f76afff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:58.000 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:57.957+0000 7f76afff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:58.000 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:57.957+0000 7f76afff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:58.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:41:58.340 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:41:58.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:58.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:58.514 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:41:58.513+0000 7f76afff77c0 -1 Falling back to public interface 2024-04-25T07:41:58.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:59.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:59.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:59.771 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:41:59.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:41:59.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:59.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:00.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:01.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:01.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:01.190 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:42:01.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:01.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:01.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:01.196 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:01.193+0000 7f76afff77c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:42:01.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:02.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:02.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:02.738 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:42:02.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:02.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:02.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:03.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:04.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:04.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:04.249 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:42:04.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:04.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:04.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:04.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:05.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:05.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:05.711 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:42:05.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:05.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:05.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:06.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:07.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:07.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:07.228 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:42:07.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:07.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:07.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:07.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:08.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:08.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:08.686 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:42:08.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:42:08.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:08.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:09.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:10.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:10.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:10.276 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:42:10.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:42:10.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:10.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:10.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:11.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:11.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:11.710 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:42:11.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:42:11.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:11.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:12.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:13.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:13.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:13.217 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:42:13.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:42:13.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:13.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:13.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:14.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:14.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:14.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:42:14.707 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:42:14.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:14.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:16.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:17.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:17.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:17.130 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:42:17.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:42:17.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:17.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:17.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:18.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:18.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:18.730 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:42:18.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:42:18.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:18.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:19.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:20.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:20.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:20.415 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:42:20.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:42:20.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:20.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:21.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:22.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:22.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:22.320 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:42:22.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:42:22.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:22.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:23.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:24.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:24.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:24.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:42:24.254 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:42:24.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:24.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:25.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:26.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:26.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:26.248 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:42:26.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:42:26.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:26.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:26.718 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1536714402,v1:127.0.0.1:6803/1536714402] [v2:127.0.0.1:6804/1536714402,v1:127.0.0.1:6805/1536714402] exists,up 0d66684e-2505-4094-84a4-6c02ac604ef5 2024-04-25T07:42:26.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:42:26.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:42:26.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:42:26.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-04-25T07:42:26.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-04-25T07:42:26.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-04-25T07:42:26.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-04-25T07:42:26.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-04-25T07:42:26.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-04-25T07:42:26.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:42:26.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-04-25T07:42:26.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-04-25T07:42:26.900 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:42:26.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-04-25T07:42:26.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:42:26.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:42:26.916 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:26.916 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:26.916 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:26.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-osd.0.asok 2024-04-25T07:42:26.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-04-25T07:42:26.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.153397/ceph-osd.0.asok log flush 2024-04-25T07:42:27.023 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:42:27.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:42:27.038 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T07:42:26.884+0000 7f76ae7b6640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-04-25T07:42:27.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:27.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153397/ceph-osd.0.asok 2024-04-25T07:42:27.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-04-25T07:42:27.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.153397/ceph-osd.0.asok log reopen 2024-04-25T07:42:27.159 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:42:27.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:42:27.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:42:27.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:27.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:42:27.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:27.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:27.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:27.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:27.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:27.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:27.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:27.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:27.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:27.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:27.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:27.287 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:27.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:27.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:27.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:27.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:27.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:42:27.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:27.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:27.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:42:27.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:27.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:27.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:27.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:27.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:27.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:27.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:27.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:27.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:27.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:27.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:27.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:27.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:27.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:27.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:27.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:27.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:42:27.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:27.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:42:27.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:27.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:27.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:42:27.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:42:27.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:27.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153397 2024-04-25T07:42:27.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:42:27.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:42:27.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:42:27.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:42:27.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-04-25T07:42:27.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-04-25T07:42:27.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-04-25T07:42:27.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-04-25T07:42:27.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-04-25T07:42:27.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-04-25T07:42:27.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-04-25T07:42:27.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-04-25T07:42:27.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-04-25T07:42:27.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-04-25T07:42:27.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-04-25T07:42:27.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-04-25T07:42:27.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-04-25T07:42:27.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-04-25T07:42:27.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:27.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:27.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:27.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:27.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:27.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:27.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:27.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:27.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:27.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:27.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:27.436 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:27.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:27.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:27.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:42:27.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:27.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:27.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:42:27.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-config 0 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:27.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:27.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:27.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:27.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:27.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:27.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:27.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:27.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:27.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:27.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:42:27.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:42:27.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:27.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153397 2024-04-25T07:42:27.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153397 2024-04-25T07:42:27.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:27.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:27.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:42:27.455 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:42:27.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:42:27.508 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-04-25T07:42:27.508 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-04-25T07:42:27.561 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-copy-from 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-04-25T07:42:27.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-04-25T07:42:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-04-25T07:42:27.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-04-25T07:42:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none ' 2024-04-25T07:42:27.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-04-25T07:42:27.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:42:27.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-04-25T07:42:27.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:27.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:27.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:27.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:27.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:27.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:27.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:27.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:27.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:27.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:27.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:27.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:27.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:42:27.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.157537 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-04-25T07:42:27.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:42:27.584 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:27.584 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:27.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.157537 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-04-25T07:42:27.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-04-25T07:42:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:42:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:42:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:42:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-04-25T07:42:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-04-25T07:42:28.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:42:28.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:28.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:28.078 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:28.078 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.078 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:28.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157537/$cluster-$name.asok' 2024-04-25T07:42:28.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:42:28.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:42:28.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:42:28.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:42:28.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:42:28.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:42:28.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:42:28.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:42:28.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:42:28.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:42:28.301 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:28.301 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.301 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:28.301 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.157537/ceph-mon.a.asok 2024-04-25T07:42:28.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:42:28.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.157537/ceph-mon.a.asok config get fsid 2024-04-25T07:42:28.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:42:28.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:42:28.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:42:28.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.157537/ceph-mon.a.asok 2024-04-25T07:42:28.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:42:28.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.157537/ceph-mon.a.asok config get mon_host 2024-04-25T07:42:28.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-04-25T07:42:28.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-04-25T07:42:28.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:42:28.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:42:28.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:42:28.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-04-25T07:42:28.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:42:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:42:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157537/$cluster-$name.asok' 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:42:29.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:42:29.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-04-25T07:42:29.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:42:29.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:42:29.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:42:29.494 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157537/$cluster-$name.asok' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:42:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0e8af53-d82b-4688-bb82-84365d4b2a64 2024-04-25T07:42:29.496 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 b0e8af53-d82b-4688-bb82-84365d4b2a64 2024-04-25T07:42:29.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b0e8af53-d82b-4688-bb82-84365d4b2a64' 2024-04-25T07:42:29.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:42:29.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlCSpmVu2YHhAA8hEIZHgwfkV2KhyWXC3r1g== 2024-04-25T07:42:29.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlCSpmVu2YHhAA8hEIZHgwfkV2KhyWXC3r1g=="}' 2024-04-25T07:42:29.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0e8af53-d82b-4688-bb82-84365d4b2a64 -i td/osd-copy-from/0/new.json 2024-04-25T07:42:29.874 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:42:29.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-04-25T07:42:29.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlCSpmVu2YHhAA8hEIZHgwfkV2KhyWXC3r1g== --osd-uuid b0e8af53-d82b-4688-bb82-84365d4b2a64 2024-04-25T07:42:29.929 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.928+0000 7f80c45747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:29.951 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.948+0000 7f80c45747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:29.953 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.948+0000 7f80c45747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:29.953 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.952+0000 7f80c45747c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:42:29.953 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.952+0000 7f80c45747c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:42:29.953 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.952+0000 7f80c45747c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:42:29.991 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:29.988+0000 7f80c45747c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-04-25T07:42:36.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-04-25T07:42:36.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:42:36.541 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:42:36.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:42:36.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:42:37.110 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:42:37.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:42:37.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:42:37.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:42:37.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:42:37.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:42:37.142 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:37.140+0000 7fb86cf357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:37.172 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:37.168+0000 7fb86cf357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:37.172+0000 7fb86cf357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:37.603 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:42:37.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:42:37.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:37.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:37.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:37.756+0000 7fb86cf357c0 -1 Falling back to public interface 2024-04-25T07:42:38.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:39.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:39.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:39.077 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:42:39.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:42:39.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:39.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:40.371 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:42:40.368+0000 7fb86cf357c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:42:40.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:40.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:40.517 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:42:40.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:40.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:40.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:41.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:42.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:42.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:42.111 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:42:42.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:42.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:42.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:42.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:43.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:43.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:43.563 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:42:43.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:43.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:43.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:44.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:45.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:45.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:45.016 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:42:45.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:45.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:45.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:46.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:46.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:46.564 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:42:46.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:46.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:46.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:47.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:48.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:48.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:48.002 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:42:48.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:42:48.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:48.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:48.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:49.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:49.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:49.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:42:49.645 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:42:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:50.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:51.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:51.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:51.278 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:42:51.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:42:51.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:51.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:51.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:52.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:52.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:52.752 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:42:52.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:42:52.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:52.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:53.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:54.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:54.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:54.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:42:54.248 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:42:54.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:54.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:55.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:56.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:56.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:56.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:42:56.621 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:42:56.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:56.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:57.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:58.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:58.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:58.787 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:42:58.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:42:58.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:58.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:59.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:00.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:00.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:00.589 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:43:00.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:43:00.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:00.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:01.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:02.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:02.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:02.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:43:02.910 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:43:02.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:02.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:03.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:04.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:04.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:04.788 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:43:04.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:43:04.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:04.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:05.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:06.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:06.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:06.406 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:43:06.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:43:06.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:06.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:06.889 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1965435443,v1:127.0.0.1:6803/1965435443] [v2:127.0.0.1:6804/1965435443,v1:127.0.0.1:6805/1965435443] exists,up b0e8af53-d82b-4688-bb82-84365d4b2a64 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-04-25T07:43:06.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:06.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157537/$cluster-$name.asok' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:43:06.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:43:06.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:43:06.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-04-25T07:43:06.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:43:06.895 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 6dcc7a47-ee63-4f8f-b8f3-fecb61078e4c 2024-04-25T07:43:06.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6dcc7a47-ee63-4f8f-b8f3-fecb61078e4c 2024-04-25T07:43:06.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6dcc7a47-ee63-4f8f-b8f3-fecb61078e4c' 2024-04-25T07:43:06.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:43:06.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKCSpmblCsNhAA92/htGBdfHyio+GcU30apA== 2024-04-25T07:43:06.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKCSpmblCsNhAA92/htGBdfHyio+GcU30apA=="}' 2024-04-25T07:43:06.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6dcc7a47-ee63-4f8f-b8f3-fecb61078e4c -i td/osd-copy-from/1/new.json 2024-04-25T07:43:07.417 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:43:07.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-04-25T07:43:07.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKCSpmblCsNhAA92/htGBdfHyio+GcU30apA== --osd-uuid 6dcc7a47-ee63-4f8f-b8f3-fecb61078e4c 2024-04-25T07:43:07.469 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.468+0000 7fcb26dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:07.471 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.472+0000 7fcb26dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:07.472 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.472+0000 7fcb26dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:07.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.476+0000 7fcb26dab7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:43:07.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.476+0000 7fcb26dab7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:43:07.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.476+0000 7fcb26dab7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:43:07.512 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:07.512+0000 7fcb26dab7c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-04-25T07:43:14.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-04-25T07:43:14.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:43:14.506 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:43:14.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:43:14.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:43:15.117 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:43:15.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:43:15.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e867dffe-4380-46e4-93e3-4239bbda7a9e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:43:15.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:43:15.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:43:15.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:43:15.148 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:15.147+0000 7f45f24237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:15.174 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:15.175+0000 7f45f24237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:15.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:15.175+0000 7f45f24237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:15.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:43:15.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:43:15.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:43:15.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:43:15.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:43:15.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:15.545 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:43:15.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:43:15.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:15.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:15.779 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:15.779+0000 7f45f24237c0 -1 Falling back to public interface 2024-04-25T07:43:16.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:17.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:17.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:17.024 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:43:17.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:43:17.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:17.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:17.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:18.451 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:43:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:43:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:18.495 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:18.495+0000 7f45f24237c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:43:18.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:19.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:19.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:19.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:43:19.890 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:43:19.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:19.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:20.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:21.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:21.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:21.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:43:21.568 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:43:21.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:21.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:22.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:43:23.051 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:43:23.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:23.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:23.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:24.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:24.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:24.550 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:43:24.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:43:24.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:24.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:25.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:26.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:26.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:26.070 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:43:26.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:43:26.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:26.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:26.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:27.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:27.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:27.601 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:43:27.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:43:27.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:27.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:28.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:29.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:29.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:29.066 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:43:29.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:43:29.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:29.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:29.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:30.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:30.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:30.570 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:43:30.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:43:30.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:30.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:31.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:32.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:32.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:32.170 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:43:32.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:43:32.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:32.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:32.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:33.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:33.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:33.741 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:43:33.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:43:33.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:33.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:34.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:35.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:35.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:35.238 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:43:35.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:43:35.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:35.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:36.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:37.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:37.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:37.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:43:37.044 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:43:37.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:37.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:38.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:39.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:39.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:39.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:43:39.314 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:43:39.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:39.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:40.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:41.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:41.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:41.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:43:41.060 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:43:41.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:41.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:41.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:42.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:42.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:42.921 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:43:42.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:43:42.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:42.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:43.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:44.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:44.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:44.829 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:43:44.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:43:44.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:44.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:45.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:46.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:46.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:46.281 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:43:46.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:43:46.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:46.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:46.772 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1009481702,v1:127.0.0.1:6811/1009481702] [v2:127.0.0.1:6812/1009481702,v1:127.0.0.1:6813/1009481702] exists,up 6dcc7a47-ee63-4f8f-b8f3-fecb61078e4c 2024-04-25T07:43:46.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:43:46.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:43:46.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:43:46.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-04-25T07:43:46.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:43:47.192 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' does not exist 2024-04-25T07:43:47.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:43:47.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:43:47.866 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' already exists 2024-04-25T07:43:47.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:43:48.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:43:51.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-04-25T07:43:51.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-04-25T07:43:51.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-04-25T07:43:51.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-04-25T07:43:51.549 INFO:tasks.workunit.client.0.smithi031.stdout:rbd/foo2 mtime 2024-04-25T07:43:51.000000+0000, size 572320 2024-04-25T07:43:51.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-04-25T07:43:51.691 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T07:43:51.721 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T07:43:51.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:43:51.775 INFO:tasks.workunit.client.0.smithi031.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-04-25T07:43:51.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:43:51.822 INFO:tasks.workunit.client.0.smithi031.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-04-25T07:43:51.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-04-25T07:43:51.962 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T07:43:51.988 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T07:43:52.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:43:52.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:43:52.310 INFO:tasks.workunit.client.0.smithi031.stdout:rbd/foo3 mtime 2024-04-25T07:43:51.000000+0000, size 572320 2024-04-25T07:43:52.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-04-25T07:43:52.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:43:52.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:43:52.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:43:52.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:52.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:52.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:52.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:52.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:52.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:52.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:52.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:52.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:52.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:52.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:52.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:52.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:52.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:52.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:43:52.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:43:52.436 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:52.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:52.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:43:52.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.157537 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-copy-from 0 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:52.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:52.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:52.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:52.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:52.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:52.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:52.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:52.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:52.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:52.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:52.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:52.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:43:52.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:43:52.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:52.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:52.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157537 2024-04-25T07:43:52.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.157537 2024-04-25T07:43:52.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:52.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:52.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:43:52.450 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:43:52.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:43:52.502 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-04-25T07:43:52.502 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-04-25T07:43:52.559 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:43:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-dup 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-04-25T07:43:52.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=8421d060-265e-41c7-89cd-caa1fc91392d --auth-supported=none ' 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs= 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-dup 0 2024-04-25T07:43:52.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:52.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:52.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:52.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:52.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:52.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:52.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:52.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:52.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:52.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:52.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:52.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:52.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:43:52.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-04-25T07:43:52.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:52.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:52.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161620 2024-04-25T07:43:52.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161620 2024-04-25T07:43:52.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:52.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:52.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:43:52.580 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:43:52.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:43:52.630 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-04-25T07:43:52.630 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-04-25T07:43:52.684 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:43:52.689 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:43:52.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:43:52.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:43:52.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:43:52.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:43:52.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-fast-mark-down 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-04-25T07:43:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-04-25T07:43:52.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-04-25T07:43:52.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-04-25T07:43:52.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-04-25T07:43:52.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none ' 2024-04-25T07:43:52.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none ' 2024-04-25T07:43:52.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-04-25T07:43:52.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-04-25T07:43:52.692 INFO:tasks.workunit.client.0.smithi031.stdout:Ensuring old behavior is there... 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:43:52.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:52.694 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:52.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:52.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:52.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:52.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:52.694 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:52.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:52.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:52.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:52.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:52.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:52.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:52.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:52.698 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:52.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:52.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:43:52.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:43:52.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:52.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:52.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:52.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161649 2024-04-25T07:43:52.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:52.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:52.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:43:52.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:43:52.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:52.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:52.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.161649 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:43:52.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:43:52.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:43:52.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:43:52.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:43:52.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:52.735+0000 7f3c0d3dac80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:43:52.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:52.735+0000 7f3c0d3dac80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:43:52.737 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:43:52.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:52.735+0000 7f3c0d3dac80 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-04-25T07:43:53.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:43:53.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:53.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:53.178 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:53.178 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:53.181 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:53.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:43:53.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:43:53.399 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:53.399+0000 7f8f3b22fc80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:43:53.399 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:53.399+0000 7f8f3b22fc80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:43:53.399 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T07:43:53.399 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:53.399+0000 7f8f3b22fc80 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-04-25T07:43:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-04-25T07:43:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-04-25T07:43:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:43:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:43:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:43:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-04-25T07:43:53.408 INFO:tasks.workunit.client.0.smithi031.stdout:Testing async msgr... 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:53.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:53.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:53.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:53.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:53.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:53.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:53.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:53.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:53.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:53.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:53.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:53.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:53.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:53.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:53.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:53.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:53.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:43:53.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:43:53.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:53.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:53.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:53.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161649 2024-04-25T07:43:53.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:53.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:53.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:43:53.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:43:53.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:53.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:53.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.161649 2024-04-25T07:43:53.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:43:53.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:43:53.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:43:53.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:43:53.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:43:53.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:53.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:53.906 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:53.907 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:53.907 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:53.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:43:53.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:43:54.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:43:54.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:43:54.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:43:54.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:43:54.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:43:54.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:43:54.128 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:43:54.128 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:43:54.128 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:43:54.129 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:43:54.129 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:54.129 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:54.130 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.161649/ceph-mon.a.asok 2024-04-25T07:43:54.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:43:54.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.161649/ceph-mon.a.asok config get fsid 2024-04-25T07:43:54.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:43:54.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:43:54.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:43:54.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:54.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.161649/ceph-mon.a.asok 2024-04-25T07:43:54.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:43:54.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.161649/ceph-mon.a.asok config get mon_host 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-04-25T07:43:54.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:55.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:43:55.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:43:55.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:43:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:43:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-04-25T07:43:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:43:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:43:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:43:55.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:43:55.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:43:55.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-04-25T07:43:55.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:43:55.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f07a2a5d-5c1e-49a7-bca4-9a830d3d18e9 2024-04-25T07:43:55.379 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 f07a2a5d-5c1e-49a7-bca4-9a830d3d18e9 2024-04-25T07:43:55.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f07a2a5d-5c1e-49a7-bca4-9a830d3d18e9' 2024-04-25T07:43:55.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:43:55.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7CSpmblTOFxAAqp8jx1kCynYGhAHwv9vW7Q== 2024-04-25T07:43:55.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7CSpmblTOFxAAqp8jx1kCynYGhAHwv9vW7Q=="}' 2024-04-25T07:43:55.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f07a2a5d-5c1e-49a7-bca4-9a830d3d18e9 -i td/osd-fast-mark-down/0/new.json 2024-04-25T07:43:55.720 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:43:55.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-04-25T07:43:55.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7CSpmblTOFxAAqp8jx1kCynYGhAHwv9vW7Q== --osd-uuid f07a2a5d-5c1e-49a7-bca4-9a830d3d18e9 2024-04-25T07:43:55.769 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.771+0000 7f871d12b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:55.797 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.799+0000 7f871d12b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:55.798 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.799+0000 7f871d12b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:55.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.799+0000 7f871d12b7c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:43:55.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.799+0000 7f871d12b7c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:43:55.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.799+0000 7f871d12b7c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:43:55.837 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:43:55.835+0000 7f871d12b7c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-04-25T07:44:02.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-04-25T07:44:02.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:02.426 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:44:02.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:44:02.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:44:03.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:44:03.002 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:44:03.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:44:03.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:44:03.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:44:03.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:44:03.037 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:03.039+0000 7f1c644d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:03.060 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:03.059+0000 7f1c644d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:03.061 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:03.059+0000 7f1c644d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:03.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:44:03.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:44:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:03.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:03.631 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:03.631+0000 7f1c644d07c0 -1 Falling back to public interface 2024-04-25T07:44:03.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:04.887 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:44:04.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:04.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:04.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:05.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:06.298 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:06.299+0000 7f1c644d07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:44:06.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:06.322 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:44:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:06.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:07.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:07.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:07.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:44:07.986 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:44:07.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:07.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:08.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:09.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:09.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:09.489 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:44:09.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:44:09.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:09.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:09.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:10.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:10.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:10.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:44:10.929 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:44:10.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:10.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:11.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:12.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:12.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:12.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:44:12.501 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:44:12.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:12.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:12.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:13.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:13.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:13.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:44:13.945 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:44:13.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:13.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:14.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:15.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:15.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:15.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:44:15.496 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:44:15.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:15.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:15.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:16.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:16.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:16.922 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:44:16.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:44:16.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:16.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:17.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:18.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:18.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:18.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:44:18.415 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:44:18.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:18.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:18.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:19.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:19.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:19.888 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:44:19.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:44:19.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:19.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:20.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:21.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:21.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:21.477 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:44:21.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:44:21.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:21.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:22.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:23.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:23.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:23.181 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:44:23.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:44:23.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:23.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:23.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:24.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:24.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:24.996 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:44:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:44:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:26.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:27.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:27.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:27.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:44:27.055 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:44:27.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:27.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:27.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:28.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:28.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:28.839 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:44:28.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:44:28.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:28.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:29.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:30.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:30.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:30.446 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:44:30.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:44:30.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:30.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:32.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:32.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:32.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:44:32.343 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:44:32.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:32.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:32.798 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2524082656,v1:127.0.0.1:6803/2524082656] [v2:127.0.0.1:6804/2524082656,v1:127.0.0.1:6805/2524082656] exists,up f07a2a5d-5c1e-49a7-bca4-9a830d3d18e9 2024-04-25T07:44:32.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:32.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:32.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:32.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=162795 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:44:32.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:32.802 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:32.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-04-25T07:44:32.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:32.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f19d50a-8ca6-481a-bb78-5a1a7a63625e 2024-04-25T07:44:32.806 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 5f19d50a-8ca6-481a-bb78-5a1a7a63625e 2024-04-25T07:44:32.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5f19d50a-8ca6-481a-bb78-5a1a7a63625e' 2024-04-25T07:44:32.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDgCSpm4IWlMRAANXUWJ5R/4KNUKJ0GBqyLnQ== 2024-04-25T07:44:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDgCSpm4IWlMRAANXUWJ5R/4KNUKJ0GBqyLnQ=="}' 2024-04-25T07:44:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f19d50a-8ca6-481a-bb78-5a1a7a63625e -i td/osd-fast-mark-down/1/new.json 2024-04-25T07:44:33.343 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:44:33.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-04-25T07:44:33.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgCSpm4IWlMRAANXUWJ5R/4KNUKJ0GBqyLnQ== --osd-uuid 5f19d50a-8ca6-481a-bb78-5a1a7a63625e 2024-04-25T07:44:33.400 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.398+0000 7f051bd297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:33.411 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.410+0000 7f051bd297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:33.412 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.410+0000 7f051bd297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:33.413 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.410+0000 7f051bd297c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:44:33.413 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.410+0000 7f051bd297c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:44:33.413 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.410+0000 7f051bd297c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:44:33.442 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:33.442+0000 7f051bd297c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-04-25T07:44:40.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-04-25T07:44:40.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:40.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:44:40.104 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:44:40.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:44:40.718 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:44:40.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:44:40.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:44:40.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:44:40.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:44:40.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:44:40.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:40.746+0000 7f87a5b7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:40.772 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:40.770+0000 7f87a5b7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:40.774+0000 7f87a5b7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:41.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:41.148 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:44:41.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:41.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:41.375 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:41.374+0000 7f87a5b7f7c0 -1 Falling back to public interface 2024-04-25T07:44:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:42.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:42.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:42.578 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:44:42.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:42.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:42.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:43.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:44.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:44.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:44.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:44.018 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:44:44.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:44.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:44.157 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:44:44.154+0000 7f87a5b7f7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:44:44.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:44:45.542 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:44:45.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:45.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:46.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:47.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:47.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:47.131 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:44:47.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:44:47.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:47.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:47.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:48.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:48.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:48.580 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:44:48.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:44:48.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:48.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:49.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:50.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:50.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:50.217 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:44:50.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:44:50.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:50.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:50.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:51.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:51.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:51.791 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:44:51.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:44:51.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:51.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:52.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:53.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:53.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:53.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:44:53.239 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:44:53.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:53.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:53.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:54.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:54.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:54.687 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:44:54.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:44:54.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:54.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:55.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:56.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:56.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:56.286 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:44:56.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:44:56.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:56.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:56.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:57.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:57.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:57.774 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:44:57.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:44:57.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:57.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:58.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:59.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:59.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:59.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:44:59.249 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:44:59.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:44:59.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:59.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:00.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:00.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:00.814 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:45:00.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:45:00.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:00.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:01.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:02.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:02.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:02.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:45:02.994 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:45:02.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:02.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:03.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:04.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:04.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:04.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:45:04.870 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:45:04.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:04.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:06.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:07.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:07.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:07.098 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:45:07.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:45:07.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:07.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:07.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:08.922 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:45:08.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:45:08.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:08.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:10.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:11.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:11.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:11.114 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:45:11.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:45:11.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:11.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:11.617 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1823457965,v1:127.0.0.1:6811/1823457965] [v2:127.0.0.1:6812/1823457965,v1:127.0.0.1:6813/1823457965] exists,up 5f19d50a-8ca6-481a-bb78-5a1a7a63625e 2024-04-25T07:45:11.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:45:11.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:45:11.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:45:11.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-04-25T07:45:11.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=164559 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:45:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-04-25T07:45:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-04-25T07:45:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:45:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:45:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:45:11.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-04-25T07:45:11.625 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:45:11.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42d5d582-cdc0-46d3-baae-84500ca4b184 2024-04-25T07:45:11.626 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 42d5d582-cdc0-46d3-baae-84500ca4b184 2024-04-25T07:45:11.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 42d5d582-cdc0-46d3-baae-84500ca4b184' 2024-04-25T07:45:11.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:45:11.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHCipmRpfZJhAAiY23fBeBoPp5gfoOh7YlpA== 2024-04-25T07:45:11.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHCipmRpfZJhAAiY23fBeBoPp5gfoOh7YlpA=="}' 2024-04-25T07:45:11.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42d5d582-cdc0-46d3-baae-84500ca4b184 -i td/osd-fast-mark-down/2/new.json 2024-04-25T07:45:12.143 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:45:12.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-04-25T07:45:12.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHCipmRpfZJhAAiY23fBeBoPp5gfoOh7YlpA== --osd-uuid 42d5d582-cdc0-46d3-baae-84500ca4b184 2024-04-25T07:45:12.197 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.194+0000 7f0b73f3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:12.264 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.262+0000 7f0b73f3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:12.265 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.266+0000 7f0b73f3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:12.266 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.266+0000 7f0b73f3e7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:45:12.266 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.266+0000 7f0b73f3e7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:45:12.266 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.266+0000 7f0b73f3e7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:45:12.388 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:12.385+0000 7f0b73f3e7c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-04-25T07:45:19.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-04-25T07:45:19.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:45:19.428 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:45:19.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:45:19.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:45:19.998 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:45:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:45:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=96a14c9a-164b-45e2-8ee1-194e4a071fd0 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161649/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:45:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:45:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:45:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:45:20.033 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:20.029+0000 7f8f0ade57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:20.062 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:20.061+0000 7f8f0ade57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:20.064 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:20.061+0000 7f8f0ade57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:20.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:45:20.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:45:20.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:45:20.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:45:20.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:45:20.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:20.481 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:45:20.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:45:20.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:20.626 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:20.625+0000 7f8f0ade57c0 -1 Falling back to public interface 2024-04-25T07:45:20.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:21.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:21.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:21.962 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:45:21.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:45:21.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:21.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:23.263 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:45:23.261+0000 7f8f0ade57c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:45:23.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:23.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:23.430 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:45:23.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:45:23.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:23.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:24.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:25.086 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:45:25.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:45:25.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:25.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:25.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:26.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:26.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:26.637 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:45:26.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:45:26.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:26.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:27.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:28.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:28.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:28.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:45:28.221 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:45:28.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:28.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:29.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:29.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:29.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:45:29.803 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:45:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:30.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:31.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:31.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:31.332 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:45:31.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:45:31.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:31.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:31.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:32.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:32.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:32.771 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:45:32.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:45:32.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:32.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:33.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:34.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:34.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:34.233 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:45:34.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:45:34.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:34.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:34.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:35.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:35.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:35.734 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:45:35.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:45:35.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:35.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:36.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:37.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:37.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:37.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:45:37.313 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:45:37.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:37.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:38.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:38.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:38.857 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:45:38.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:45:38.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:38.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:39.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:40.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:40.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:40.950 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:45:40.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:45:40.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:40.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:41.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:42.934 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:45:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:45:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:44.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:45.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:45.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:45.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:45:45.151 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:45:45.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:45.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:46.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:47.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:47.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:47.138 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:45:47.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:45:47.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:47.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:48.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:49.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:49.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:49.043 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:45:49.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:45:49.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:49.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:45:49.669 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1772251791,v1:127.0.0.1:6819/1772251791] [v2:127.0.0.1:6820/1772251791,v1:127.0.0.1:6821/1772251791] exists,up 42d5d582-cdc0-46d3-baae-84500ca4b184 2024-04-25T07:45:49.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:45:49.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:45:49.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:45:49.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-04-25T07:45:49.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=166326 2024-04-25T07:45:49.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-04-25T07:45:49.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:45:50.093 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' does not exist 2024-04-25T07:45:50.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:45:50.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:45:50.719 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' already exists 2024-04-25T07:45:50.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:45:51.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:45:54.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-04-25T07:45:54.317 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T07:45:54.317 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-04-25T07:45:54.317 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_167296 2024-04-25T07:45:54.317 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:45:54.317 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:45:55.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 66 50 0.195289 0.195312 0.250528 0.246114 2024-04-25T07:45:56.317 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 128 112 0.218723 0.242188 0.30298 0.277375 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 2.23566 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 128 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 4096 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 4096 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.223647 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0331456 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.242188 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.195312 2024-04-25T07:45:57.318 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 57 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 8.48528 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 62 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 50 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.277786 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0328052 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.35146 2024-04-25T07:45:57.319 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.114646 2024-04-25T07:45:57.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-04-25T07:45:58.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=162795 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 162795 -eq 0 ']' 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=162795 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 162795 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:45:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:45:58.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:45:59.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:45:59.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:45:59.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:45:59.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:45:59.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:45:59.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-04-25T07:45:59.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 162795 -eq 162795 ']' 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164559 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164559 -eq 162795 ']' 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=164559 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 164559 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:45:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:46:00.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:46:00.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:46:00.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:46:00.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:46:00.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:46:01.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-04-25T07:46:01.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:46:01.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-04-25T07:46:01.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:46:01.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:46:01.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-04-25T07:46:01.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:01.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:01.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:01.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:01.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:01.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:01.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:01.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:01.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:01.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:01.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:01.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:01.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:01.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:01.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:01.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:46:01.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:01.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:01.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:46:01.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161649 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-fast-mark-down 0 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:46:01.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:01.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:01.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:01.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:01.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:01.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:01.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:01.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:01.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:01.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:01.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:01.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:01.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:01.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:01.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:01.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:01.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:01.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:46:01.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:46:01.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:01.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:01.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161649 2024-04-25T07:46:01.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161649 2024-04-25T07:46:01.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:01.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:01.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:46:01.556 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:46:01.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:46:01.607 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-04-25T07:46:01.607 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-04-25T07:46:01.659 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:46:01.663 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-force-create-pg 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-04-25T07:46:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-04-25T07:46:01.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-04-25T07:46:01.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none ' 2024-04-25T07:46:01.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-04-25T07:46:01.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-04-25T07:46:01.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:46:01.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-04-25T07:46:01.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:01.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:01.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:01.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:01.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:01.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:01.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:01.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:01.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:01.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:01.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:01.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:01.673 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:01.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:01.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:01.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:46:01.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:01.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:01.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:01.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.167427 2024-04-25T07:46:01.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:01.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:01.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-04-25T07:46:01.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:46:01.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:01.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:01.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.167427 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:46:01.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:46:01.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-04-25T07:46:01.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:46:02.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:46:02.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:02.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:02.187 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:02.187 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:02.187 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:02.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:46:02.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:46:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:46:02.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:46:02.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:46:02.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:46:02.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.167427/ceph-mon.a.asok 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.167427/ceph-mon.a.asok config get fsid 2024-04-25T07:46:02.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:46:02.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:46:02.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:46:02.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.167427/ceph-mon.a.asok 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:46:02.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.167427/ceph-mon.a.asok config get mon_host 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-04-25T07:46:02.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:46:03.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:46:03.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:03.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:03.608 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:03.608 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:03.608 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:03.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:46:03.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:46:03.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:46:03.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:46:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:46:03.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:46:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:46:03.649 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:46:03.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb891e01-81c7-40d8-9c4f-142e552682c4 2024-04-25T07:46:03.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eb891e01-81c7-40d8-9c4f-142e552682c4' 2024-04-25T07:46:03.649 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 eb891e01-81c7-40d8-9c4f-142e552682c4 2024-04-25T07:46:03.649 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:46:03.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7CipmWSqzJxAAMjDY1Iyv5yCGYOZkGysIeQ== 2024-04-25T07:46:03.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7CipmWSqzJxAAMjDY1Iyv5yCGYOZkGysIeQ=="}' 2024-04-25T07:46:03.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb891e01-81c7-40d8-9c4f-142e552682c4 -i td/osd-force-create-pg/0/new.json 2024-04-25T07:46:03.951 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:46:03.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-04-25T07:46:03.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7CipmWSqzJxAAMjDY1Iyv5yCGYOZkGysIeQ== --osd-uuid eb891e01-81c7-40d8-9c4f-142e552682c4 2024-04-25T07:46:03.992 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:03.989+0000 7fc9eeb887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:04.011 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:04.009+0000 7fc9eeb887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:04.012 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:04.009+0000 7fc9eeb887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:04.013 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:04.009+0000 7fc9eeb887c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:46:04.013 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:04.009+0000 7fc9eeb887c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:46:04.013 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:04.009+0000 7fc9eeb887c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:46:04.052 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:04.049+0000 7fc9eeb887c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-04-25T07:46:10.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-04-25T07:46:10.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:46:10.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:46:10.575 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:46:10.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:46:11.143 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:46:11.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:46:11.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:46:11.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:46:11.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:46:11.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:46:11.174 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:11.173+0000 7f487af597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:11.173+0000 7f487af597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:11.177 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:11.177+0000 7f487af597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:11.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:46:11.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:46:11.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:46:11.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:46:11.668 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:46:11.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:46:11.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:11.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:46:11.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:11.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:11.735 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:11.733+0000 7f487af597c0 -1 Falling back to public interface 2024-04-25T07:46:12.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:13.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:13.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:13.142 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:46:13.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:46:13.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:13.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:13.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:14.334 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:14.333+0000 7f487af597c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:46:14.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:14.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:14.582 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:46:14.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:46:14.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:14.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:15.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:16.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:16.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:16.078 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:46:16.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:46:16.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:16.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:16.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:17.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:17.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:17.676 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:46:17.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:46:17.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:17.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:18.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:19.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:19.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:19.228 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:46:19.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:46:19.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:19.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:19.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:20.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:20.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:20.755 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:46:20.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:46:20.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:20.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:21.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:22.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:22.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:22.340 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:46:22.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:46:22.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:22.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:22.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:23.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:23.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:23.837 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:46:23.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:46:23.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:23.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:24.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:25.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:25.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:25.405 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:46:25.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:46:25.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:25.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:25.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:26.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:26.852 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:46:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:46:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:27.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:28.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:28.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:28.318 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:46:28.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:46:28.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:28.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:29.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:30.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:30.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:30.104 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:46:30.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:46:30.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:30.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:30.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:31.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:31.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:31.597 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:46:31.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:46:31.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:31.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:32.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:33.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:33.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:33.409 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:46:33.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:46:33.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:33.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:34.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:35.214 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:46:35.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:46:35.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:35.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:36.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:37.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:37.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:37.395 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:46:37.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:46:37.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:37.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:38.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:39.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:39.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:39.302 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:46:39.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:46:39.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:39.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:39.857 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37257112,v1:127.0.0.1:6803/37257112] [v2:127.0.0.1:6804/37257112,v1:127.0.0.1:6805/37257112] exists,up eb891e01-81c7-40d8-9c4f-142e552682c4 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:46:39.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:46:39.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:46:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:46:39.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:46:39.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6da2eca0-8316-4127-8f2e-f7892edae3e2 2024-04-25T07:46:39.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6da2eca0-8316-4127-8f2e-f7892edae3e2' 2024-04-25T07:46:39.863 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 6da2eca0-8316-4127-8f2e-f7892edae3e2 2024-04-25T07:46:39.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:46:39.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfCipmo6jZNBAA2WMmg5BsD8EM3MOqn8jUmg== 2024-04-25T07:46:39.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfCipmo6jZNBAA2WMmg5BsD8EM3MOqn8jUmg=="}' 2024-04-25T07:46:39.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6da2eca0-8316-4127-8f2e-f7892edae3e2 -i td/osd-force-create-pg/1/new.json 2024-04-25T07:46:40.406 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:46:40.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-04-25T07:46:40.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfCipmo6jZNBAA2WMmg5BsD8EM3MOqn8jUmg== --osd-uuid 6da2eca0-8316-4127-8f2e-f7892edae3e2 2024-04-25T07:46:40.458 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.456+0000 7f11b80737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:40.516 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.512+0000 7f11b80737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:40.518 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.516+0000 7f11b80737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:40.518 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.516+0000 7f11b80737c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:46:40.518 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.516+0000 7f11b80737c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:46:40.518 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.516+0000 7f11b80737c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:46:40.640 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:40.640+0000 7f11b80737c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-04-25T07:46:47.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-04-25T07:46:47.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:46:47.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:46:47.410 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:46:47.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:46:47.967 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:46:47.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:46:47.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:46:47.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:46:47.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:46:47.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:46:47.997 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:47.996+0000 7fc3bb03a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:47.999 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:47.996+0000 7fc3bb03a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:48.001 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:47.996+0000 7fc3bb03a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:46:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:46:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:46:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:46:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:46:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:46:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:46:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:48.560 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:48.556+0000 7fc3bb03a7c0 -1 Falling back to public interface 2024-04-25T07:46:48.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:49.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:49.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:49.882 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:46:49.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:46:49.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:49.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:50.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:46:51.208+0000 7fc3bb03a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:46:51.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:51.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:51.318 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:46:51.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:46:51.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:51.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:52.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:53.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:53.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:53.003 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:46:53.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:46:53.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:53.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:53.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:54.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:54.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:54.582 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:46:54.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:46:54.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:54.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:55.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:56.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:56.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:56.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:46:56.173 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:46:56.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:56.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:57.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:57.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:57.630 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:46:57.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:46:57.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:57.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:58.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:59.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:59.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:59.139 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:46:59.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:46:59.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:59.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:59.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:00.680 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:47:00.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:47:00.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:00.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:01.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:02.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:02.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:02.117 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:47:02.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:47:02.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:02.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:02.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:03.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:03.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:03.672 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:47:03.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:47:03.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:03.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:04.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:05.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:05.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:05.274 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:47:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:47:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:05.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:06.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:06.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:06.909 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:47:06.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:47:06.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:06.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:08.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:09.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:09.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:09.124 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:47:09.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:47:09.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:09.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:10.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:11.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:11.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:11.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:47:11.018 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:47:11.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:11.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:12.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:13.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:13.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:13.240 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:47:13.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:47:13.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:13.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:13.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:14.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:14.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:14.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:47:14.949 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:47:14.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:14.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:15.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:16.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:16.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:16.844 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:47:16.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:47:16.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:16.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:17.271 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4048714067,v1:127.0.0.1:6811/4048714067] [v2:127.0.0.1:6812/4048714067,v1:127.0.0.1:6813/4048714067] exists,up 6da2eca0-8316-4127-8f2e-f7892edae3e2 2024-04-25T07:47:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:47:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:47:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:47:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:47:17.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:47:17.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:47:17.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f40bc013-2682-45f4-9888-00aa0afddb49 2024-04-25T07:47:17.278 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 f40bc013-2682-45f4-9888-00aa0afddb49 2024-04-25T07:47:17.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f40bc013-2682-45f4-9888-00aa0afddb49' 2024-04-25T07:47:17.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:47:17.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFCipmx/TiERAAxJRmQMszAMEnUy7CQmIBMg== 2024-04-25T07:47:17.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFCipmx/TiERAAxJRmQMszAMEnUy7CQmIBMg=="}' 2024-04-25T07:47:17.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f40bc013-2682-45f4-9888-00aa0afddb49 -i td/osd-force-create-pg/2/new.json 2024-04-25T07:47:17.815 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:47:17.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-04-25T07:47:17.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFCipmx/TiERAAxJRmQMszAMEnUy7CQmIBMg== --osd-uuid f40bc013-2682-45f4-9888-00aa0afddb49 2024-04-25T07:47:17.865 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.867+0000 7f0fe60077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:17.902 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.903+0000 7f0fe60077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:17.904 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.903+0000 7f0fe60077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:17.904 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.903+0000 7f0fe60077c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:47:17.904 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.903+0000 7f0fe60077c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:47:17.905 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.903+0000 7f0fe60077c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:47:17.949 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:17.947+0000 7f0fe60077c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-04-25T07:47:24.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-04-25T07:47:24.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:47:24.899 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:47:24.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:47:24.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:47:25.459 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:47:25.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:47:25.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:47:25.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:47:25.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:47:25.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:47:25.492 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:25.491+0000 7f498be097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:25.494 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:25.495+0000 7f498be097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:25.495 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:25.495+0000 7f498be097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:25.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:47:25.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:47:25.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:47:25.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:47:25.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:47:25.930 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:47:25.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:25.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:47:25.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:25.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:26.090 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:26.091+0000 7f498be097c0 -1 Falling back to public interface 2024-04-25T07:47:26.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:27.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:27.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:27.354 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:47:27.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:47:27.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:27.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:27.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:28.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:28.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:28.835 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:47:28.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:47:28.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:28.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:28.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:47:28.879+0000 7f498be097c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:47:29.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:30.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:30.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:30.280 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:47:30.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:47:30.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:30.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:30.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:31.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:31.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:31.719 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:47:31.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:47:31.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:31.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:32.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:33.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:33.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:33.144 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:47:33.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:47:33.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:33.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:33.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:34.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:34.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:34.646 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:47:34.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:47:34.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:34.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:35.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:36.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:36.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:36.237 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:47:36.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:47:36.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:36.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:36.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:37.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:37.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:37.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:47:37.671 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:47:37.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:37.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:38.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:39.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:39.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:39.124 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:47:39.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:47:39.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:39.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:39.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:40.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:40.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:40.692 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:47:40.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:47:40.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:40.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:41.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:42.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:42.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:42.293 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:47:42.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:47:42.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:42.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:42.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:43.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:43.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:43.822 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:47:43.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:47:43.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:43.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:44.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:45.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:45.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:45.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:47:45.576 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:47:45.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:45.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:46.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:47.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:47.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:47.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:47:47.606 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:47:47.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:47.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:48.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:49.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:49.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:49.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:47:49.499 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:47:49.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:49.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:50.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:51.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:51.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:51.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:47:51.425 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:47:51.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:51.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:52.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:53.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:53.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:53.592 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:47:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:47:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:54.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:55.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:55.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:55.190 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:47:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:47:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:47:55.626 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4232747660,v1:127.0.0.1:6819/4232747660] [v2:127.0.0.1:6820/4232747660,v1:127.0.0.1:6821/4232747660] exists,up f40bc013-2682-45f4-9888-00aa0afddb49 2024-04-25T07:47:55.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:47:55.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:47:55.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:47:55.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-04-25T07:47:56.166 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T07:47:56.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-04-25T07:47:56.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:47:56.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:47:56.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:47:56.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:47:56.184 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:47:56.184 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:47:56.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:47:56.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:47:56.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:47:56.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:47:56.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:47:56.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:47:56.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:47:56.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:47:56.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:47:56.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:47:56.978 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:47:56.978 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:47:56.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:47:56.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:47:56.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:47:57.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-04-25T07:47:57.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-04-25T07:47:57.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-04-25T07:47:57.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:47:57.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:47:57.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T07:47:57.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T07:47:57.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-04-25T07:47:57.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:47:57.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:47:57.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T07:47:57.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T07:47:57.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509449' 2024-04-25T07:47:57.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:47:57.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-04-25T07:47:57.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:47:57.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:47:57.649 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-04-25T07:47:57.649 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:47:57.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-04-25T07:47:57.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-25T07:47:57.650 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836505 2024-04-25T07:47:57.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:47:58.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836505 2024-04-25T07:47:58.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:47:59.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:47:59.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:47:59.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-25T07:47:59.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:47:59.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T07:47:59.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:47:59.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:47:59.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T07:47:59.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:47:59.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T07:47:59.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T07:47:59.640 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672977 2024-04-25T07:47:59.640 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:48:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672977 2024-04-25T07:48:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:48:00.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T07:48:00.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:48:00.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:48:00.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:48:00.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T07:48:00.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T07:48:00.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T07:48:00.077 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509449 2024-04-25T07:48:00.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:48:00.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-25T07:48:00.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:48:00.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:00.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:01.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:48:01.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:01.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:01.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:01.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:01.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:01.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:01.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:01.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:48:01.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:01.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:01.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:02.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:48:02.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:48:02.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:48:02.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-04-25T07:48:02.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:02.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:02.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:02.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:02.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:02.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:02.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-04-25T07:48:02.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:02.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:02.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:02.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:02.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:03.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:03.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-04-25T07:48:03.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:03.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:03.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:03.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:03.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:04.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:04.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-04-25T07:48:05.490 INFO:tasks.workunit.client.0.smithi031.stderr:PG '1.0' not found 2024-04-25T07:48:06.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-04-25T07:48:07.470 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T07:48:07.470 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T07:48:07.470 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:48:07.602 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T07:48:08.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-04-25T07:48:09.450 INFO:tasks.workunit.client.0.smithi031.stderr:PG '1.0' not found 2024-04-25T07:48:09.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-04-25T07:48:09.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:48:09.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:48:09.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:48:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:48:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:48:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:48:09.993 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:48:09.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:09.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-04-25T07:48:09.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:48:09.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:48:09.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:48:09.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:48:10.028 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:10.026+0000 7ff0d78ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:10.031 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:10.030+0000 7ff0d78ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:10.032 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:10.030+0000 7ff0d78ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:10.428 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:48:10.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:10.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:10.595 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:10.594+0000 7ff0d78ea7c0 -1 Falling back to public interface 2024-04-25T07:48:10.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:11.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:11.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:11.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:48:11.912 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:48:11.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:11.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:12.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:13.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:13.335 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:48:13.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:13.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:48:13.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:13.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:13.337 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:13.334+0000 7ff0d78ea7c0 -1 osd.0 23 log_to_monitors true 2024-04-25T07:48:13.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:14.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:14.802 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:48:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:48:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:14.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:15.224 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 28 up_thru 28 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2710058882,v1:127.0.0.1:6803/2710058882] [v2:127.0.0.1:6804/2710058882,v1:127.0.0.1:6805/2710058882] exists,up eb891e01-81c7-40d8-9c4f-142e552682c4 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:48:15.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:48:15.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:48:15.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:48:15.231 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:48:15.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:48:15.257 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:15.258+0000 7faeb0a5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:15.259 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:15.258+0000 7faeb0a5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:15.260 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:15.258+0000 7faeb0a5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:15.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:48:15.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:15.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:48:15.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:15.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:15.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:15.657 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:48:15.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:15.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:15.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:15.826+0000 7faeb0a5f7c0 -1 Falling back to public interface 2024-04-25T07:48:16.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:17.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:17.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:17.081 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:48:17.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:48:17.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:17.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:17.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:18.517 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:48:18.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:48:18.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:18.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:18.672 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:18.670+0000 7faeb0a5f7c0 -1 osd.1 24 log_to_monitors true 2024-04-25T07:48:18.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:19.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:19.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:19.998 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:48:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:48:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:19.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:20.437 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 25 last_clean_interval [10,24) [v2:127.0.0.1:6810/866333695,v1:127.0.0.1:6811/866333695] [v2:127.0.0.1:6812/866333695,v1:127.0.0.1:6813/866333695] exists,up 6da2eca0-8316-4127-8f2e-f7892edae3e2 2024-04-25T07:48:20.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:48:20.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:48:20.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:48:20.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:20.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:20.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:20.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:20.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:20.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:48:20.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:48:20.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:48:20.443 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:48:20.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:48:20.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c9850233-689f-4297-9124-9e58df6e67a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167427/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:20.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-04-25T07:48:20.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:48:20.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:48:20.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:48:20.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:48:20.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:20.470+0000 7f49f92c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:20.502 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:20.502+0000 7f49f92c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:20.503 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:20.502+0000 7f49f92c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:48:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:48:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:20.885 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:48:20.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:20.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:20.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:20.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:21.051 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:21.050+0000 7f49f92c17c0 -1 Falling back to public interface 2024-04-25T07:48:21.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:22.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:22.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:22.308 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:48:22.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:48:22.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:22.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:22.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:23.699 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:23.698+0000 7f49f92c17c0 -1 osd.2 25 log_to_monitors true 2024-04-25T07:48:23.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:23.743 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:48:23.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:23.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:48:23.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:23.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:24.325 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 34 up_thru 17 down_at 26 last_clean_interval [15,25) [v2:127.0.0.1:6818/1607558731,v1:127.0.0.1:6819/1607558731] [v2:127.0.0.1:6820/1607558731,v1:127.0.0.1:6821/1607558731] exists,up f40bc013-2682-45f4-9888-00aa0afddb49 2024-04-25T07:48:24.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:48:24.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:48:24.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:48:24.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-04-25T07:48:34.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-04-25T07:48:34.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-04-25T07:48:34.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-04-25T07:48:34.795 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 37s 0'0 19:8 [1]p1 [1]p1 2024-04-25T07:47:56.000372+0000 2024-04-25T07:47:56.000372+0000 0 no scrub is scheduled 2024-04-25T07:48:34.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-04-25T07:48:35.479 INFO:tasks.workunit.client.0.smithi031.stderr:pg 1.0 already creating 2024-04-25T07:48:35.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-04-25T07:48:35.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:48:35.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:48:35.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:48:35.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:48:35.501 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:48:35.501 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:48:35.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:48:35.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:48:35.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:48:35.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:48:35.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:48:35.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:48:35.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:48:35.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:48:35.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:48:36.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:48:36.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084295 2024-04-25T07:48:36.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084295 2024-04-25T07:48:36.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-120259084295' 2024-04-25T07:48:36.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:48:36.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:48:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=133143986182 2024-04-25T07:48:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 133143986182 2024-04-25T07:48:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-120259084295 1-133143986182' 2024-04-25T07:48:36.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:48:36.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:48:36.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888069 2024-04-25T07:48:36.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888069 2024-04-25T07:48:36.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-120259084295 1-133143986182 2-146028888069' 2024-04-25T07:48:36.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:48:36.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-120259084295 2024-04-25T07:48:36.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:48:36.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:48:36.589 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-120259084295 2024-04-25T07:48:36.589 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:48:36.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084295 2024-04-25T07:48:36.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-04-25T07:48:36.591 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 120259084295 2024-04-25T07:48:36.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:48:37.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084295 -lt 120259084295 2024-04-25T07:48:37.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:48:37.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-133143986182 2024-04-25T07:48:37.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:48:37.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:48:37.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-133143986182 2024-04-25T07:48:37.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:48:37.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=133143986182 2024-04-25T07:48:37.033 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 133143986182 2024-04-25T07:48:37.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 133143986182' 2024-04-25T07:48:37.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:48:37.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 133143986180 -lt 133143986182 2024-04-25T07:48:37.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:48:38.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:48:38.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:48:38.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 133143986182 -lt 133143986182 2024-04-25T07:48:38.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:48:38.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888069 2024-04-25T07:48:38.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:48:38.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:48:38.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888069 2024-04-25T07:48:38.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:48:38.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888069 2024-04-25T07:48:38.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888069' 2024-04-25T07:48:38.997 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 146028888069 2024-04-25T07:48:38.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:48:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888069 -lt 146028888069 2024-04-25T07:48:39.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:48:39.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:39.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:40.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:40.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:40.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:48:40.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:40.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:40.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:41.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:48:41.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:48:41.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:48:41.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-04-25T07:48:41.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:41.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:41.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:41.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:41.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:41.349 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:41.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:41.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:41.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:41.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:41.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:41.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:41.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:41.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:41.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:48:41.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:48:41.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:41.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:41.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:48:41.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.167427 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-force-create-pg 0 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:48:41.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:41.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:41.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:41.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:41.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:41.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:41.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:41.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:41.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:41.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:41.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:41.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:41.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:41.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:41.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:41.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:41.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:41.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:48:41.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:48:41.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:41.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:41.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167427 2024-04-25T07:48:41.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.167427 2024-04-25T07:48:41.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:41.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:41.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:48:41.393 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:48:41.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:48:41.453 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-04-25T07:48:41.454 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-04-25T07:48:41.511 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:48:41.516 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:48:41.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-markdown 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-04-25T07:48:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-04-25T07:48:41.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-04-25T07:48:41.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none ' 2024-04-25T07:48:41.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-04-25T07:48:41.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-04-25T07:48:41.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:48:41.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-04-25T07:48:41.520 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_markdown_boot_exceed_time 2024-04-25T07:48:41.520 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_markdown_exceed_maxdown_count 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_osd_stop' 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:48:41.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:48:41.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:41.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:41.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:41.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:41.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:41.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:41.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:41.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:41.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:41.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:41.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:41.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:41.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:41.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:41.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:41.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:41.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:41.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:48:41.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:48:41.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:41.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:41.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:41.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:48:41.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:41.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:41.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:48:41.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:48:41.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:41.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:41.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176567 2024-04-25T07:48:41.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:48:41.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:48:42.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:48:42.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:42.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:42.017 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:42.017 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:42.017 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:42.017 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:48:42.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:48:42.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:48:42.246 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:48:42.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:48:42.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:48:42.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:48:42.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:48:42.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get fsid 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:48:42.365 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:48:42.366 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:48:42.366 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:42.366 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:42.366 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:48:42.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:48:42.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get mon_host 2024-04-25T07:48:42.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-04-25T07:48:42.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:48:42.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:48:42.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:48:42.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:48:42.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:48:42.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:43.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:48:43.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:48:43.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:48:43.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-04-25T07:48:43.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:48:43.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:48:43.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:48:43.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:43.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:43.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:48:43.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:48:43.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:48:43.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:48:43.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=859ff0c2-1f0a-42bd-97ed-6c5ca3dc8f83 2024-04-25T07:48:43.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 859ff0c2-1f0a-42bd-97ed-6c5ca3dc8f83' 2024-04-25T07:48:43.401 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 859ff0c2-1f0a-42bd-97ed-6c5ca3dc8f83 2024-04-25T07:48:43.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:48:43.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbCipms2XfGBAAhRa1ZvAvd2wVGbrKOBjxOw== 2024-04-25T07:48:43.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbCipms2XfGBAAhRa1ZvAvd2wVGbrKOBjxOw=="}' 2024-04-25T07:48:43.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 859ff0c2-1f0a-42bd-97ed-6c5ca3dc8f83 -i td/osd-markdown/0/new.json 2024-04-25T07:48:43.722 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:48:43.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:48:43.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbCipms2XfGBAAhRa1ZvAvd2wVGbrKOBjxOw== --osd-uuid 859ff0c2-1f0a-42bd-97ed-6c5ca3dc8f83 2024-04-25T07:48:43.775 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.774+0000 7eff19d727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:43.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.798+0000 7eff19d727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:43.800 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.798+0000 7eff19d727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:43.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.798+0000 7eff19d727c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:48:43.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.798+0000 7eff19d727c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:48:43.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.798+0000 7eff19d727c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:48:43.831 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:43.830+0000 7eff19d727c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:48:50.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:48:50.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:48:50.424 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:48:50.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:48:50.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:48:50.996 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:48:50.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:48:50.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:48:50.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:50.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:48:50.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:48:51.027 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:51.026+0000 7f8c2e5157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:51.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:51.054+0000 7f8c2e5157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:51.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:51.054+0000 7f8c2e5157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:51.495 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:48:51.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:51.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:51.616 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:51.614+0000 7f8c2e5157c0 -1 Falling back to public interface 2024-04-25T07:48:51.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:52.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:52.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:52.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:48:52.924 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:48:52.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:52.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:53.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:54.328 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:48:54.326+0000 7f8c2e5157c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:48:54.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:54.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:54.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:48:54.348 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:48:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:54.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:55.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:55.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:55.863 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:48:55.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:48:55.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:55.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:56.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:57.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:57.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:57.305 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:48:57.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:48:57.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:57.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:57.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:58.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:58.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:58.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:48:58.764 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:48:58.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:58.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:59.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:00.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:00.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:00.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:49:00.347 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:49:00.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:00.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:00.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:01.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:49:01.784 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:49:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:02.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:49:03.365 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:49:03.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:03.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:03.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:04.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:04.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:04.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:49:04.832 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:49:04.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:04.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:05.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:06.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:06.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:06.260 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:49:06.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:49:06.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:06.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:06.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:07.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:07.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:07.710 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:49:07.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:49:07.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:07.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:08.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:09.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:09.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:09.307 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:49:09.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:49:09.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:09.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:09.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:10.812 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:49:10.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:10.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:10.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:49:10.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:10.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:12.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:13.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:13.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:13.019 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:49:13.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:49:13.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:13.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:13.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:14.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:14.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:14.871 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:49:14.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:49:14.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:14.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:16.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:17.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:17.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:17.110 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:49:17.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:49:17.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:17.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:17.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:18.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:18.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:18.887 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:49:18.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:49:18.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:18.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:19.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:20.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:20.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:20.928 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:49:20.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:49:20.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:20.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:21.358 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/618514979,v1:127.0.0.1:6803/618514979] [v2:127.0.0.1:6804/618514979,v1:127.0.0.1:6805/618514979] exists,up 859ff0c2-1f0a-42bd-97ed-6c5ca3dc8f83 2024-04-25T07:49:21.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:49:21.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:49:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:49:21.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:49:21.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:49:21.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f167f59-e0b5-4eac-87c2-eb6649b8a9fe 2024-04-25T07:49:21.365 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 7f167f59-e0b5-4eac-87c2-eb6649b8a9fe 2024-04-25T07:49:21.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7f167f59-e0b5-4eac-87c2-eb6649b8a9fe' 2024-04-25T07:49:21.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:49:21.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABCypmddwyFxAA6xjG4jP4sIZZIzC3pBijOg== 2024-04-25T07:49:21.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABCypmddwyFxAA6xjG4jP4sIZZIzC3pBijOg=="}' 2024-04-25T07:49:21.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f167f59-e0b5-4eac-87c2-eb6649b8a9fe -i td/osd-markdown/1/new.json 2024-04-25T07:49:21.895 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:49:21.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:49:21.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABCypmddwyFxAA6xjG4jP4sIZZIzC3pBijOg== --osd-uuid 7f167f59-e0b5-4eac-87c2-eb6649b8a9fe 2024-04-25T07:49:21.944 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:21.941+0000 7f0b194907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:21.975 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:21.973+0000 7f0b194907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:21.976 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:21.973+0000 7f0b194907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:21.977 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:21.973+0000 7f0b194907c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:49:21.977 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:21.973+0000 7f0b194907c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:49:21.977 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:21.973+0000 7f0b194907c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:49:22.006 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:22.005+0000 7f0b194907c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:49:28.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:49:28.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:49:28.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:49:28.653 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:49:28.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:49:29.209 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:49:29.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:49:29.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:49:29.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:49:29.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:49:29.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:49:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:29.237+0000 7f10a2b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:29.241 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:29.241+0000 7f10a2b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:29.242 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:29.241+0000 7f10a2b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:29.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:49:29.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:49:29.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:49:29.672 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:49:29.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:49:29.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:49:29.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:29.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:49:29.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:29.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:29.798 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:29.797+0000 7f10a2b177c0 -1 Falling back to public interface 2024-04-25T07:49:30.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:31.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:31.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:31.113 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:49:31.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:49:31.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:31.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:31.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:32.493 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:49:32.489+0000 7f10a2b177c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:49:32.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:32.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:32.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:49:32.539 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:49:32.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:32.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:33.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:34.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:34.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:34.129 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:49:34.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:49:34.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:34.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:34.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:35.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:35.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:35.751 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:49:35.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:49:35.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:35.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:37.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:37.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:37.213 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:49:37.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:49:37.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:37.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:37.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:38.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:38.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:38.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:49:38.665 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:49:38.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:38.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:39.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:40.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:40.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:40.165 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:49:40.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:49:40.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:40.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:40.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:41.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:41.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:41.735 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:49:41.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:49:41.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:41.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:42.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:43.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:43.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:43.248 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:49:43.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:49:43.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:43.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:43.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:44.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:44.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:44.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:49:44.809 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:49:44.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:44.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:45.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:46.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:46.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:46.338 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:49:46.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:49:46.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:46.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:47.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:48.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:48.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:48.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:49:48.077 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:49:48.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:48.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:48.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:49.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:49.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:49.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:49:49.690 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:49:49.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:49.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:50.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:51.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:51.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:51.527 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:49:51.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:49:51.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:51.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:52.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:53.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:53.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:53.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:49:53.349 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:49:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:54.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:55.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:55.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:55.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:49:55.183 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:49:55.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:55.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:56.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:57.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:57.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:57.275 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:49:57.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:49:57.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:57.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:58.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:59.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:59.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:59.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:49:59.201 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:49:59.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:59.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:59.675 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1938293573,v1:127.0.0.1:6811/1938293573] [v2:127.0.0.1:6812/1938293573,v1:127.0.0.1:6813/1938293573] exists,up 7f167f59-e0b5-4eac-87c2-eb6649b8a9fe 2024-04-25T07:49:59.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:49:59.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:49:59.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:49:59.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-04-25T07:49:59.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:49:59.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:49:59.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:49:59.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:49:59.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:49:59.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7669964f-875d-4c3a-9f9c-1534a6ac4bf4 2024-04-25T07:49:59.682 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 7669964f-875d-4c3a-9f9c-1534a6ac4bf4 2024-04-25T07:49:59.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7669964f-875d-4c3a-9f9c-1534a6ac4bf4' 2024-04-25T07:49:59.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:49:59.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnCypmO+ypKRAA0f9CwvSF3QYvq7c8ZbjeCQ== 2024-04-25T07:49:59.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnCypmO+ypKRAA0f9CwvSF3QYvq7c8ZbjeCQ=="}' 2024-04-25T07:49:59.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7669964f-875d-4c3a-9f9c-1534a6ac4bf4 -i td/osd-markdown/2/new.json 2024-04-25T07:50:00.199 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:50:00.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:50:00.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnCypmO+ypKRAA0f9CwvSF3QYvq7c8ZbjeCQ== --osd-uuid 7669964f-875d-4c3a-9f9c-1534a6ac4bf4 2024-04-25T07:50:00.243 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.241+0000 7f320e2c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:00.302 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.301+0000 7f320e2c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:00.304 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.301+0000 7f320e2c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:00.304 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.301+0000 7f320e2c67c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:50:00.304 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.301+0000 7f320e2c67c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:50:00.304 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.301+0000 7f320e2c67c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:50:00.417 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:00.417+0000 7f320e2c67c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:50:07.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:50:07.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:50:07.600 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:50:07.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:50:07.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:50:08.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:50:08.176 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:50:08.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:50:08.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:50:08.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:50:08.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:50:08.206 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:08.204+0000 7f493e4147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:08.234 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:08.232+0000 7f493e4147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:08.236 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:08.232+0000 7f493e4147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:08.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:50:08.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:50:08.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:50:08.618 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:50:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:50:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:50:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:50:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:08.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:08.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:08.796+0000 7f493e4147c0 -1 Falling back to public interface 2024-04-25T07:50:09.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:10.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:10.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:10.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:50:10.020 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:50:10.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:10.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:10.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:11.401 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:50:11.396+0000 7f493e4147c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:50:11.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:11.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:11.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:50:11.454 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:50:11.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:11.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:13.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:13.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:13.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:50:13.055 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:50:13.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:13.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:13.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:14.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:14.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:14.547 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:50:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:50:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:15.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:16.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:16.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:16.022 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:50:16.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:50:16.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:16.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:16.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:17.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:17.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:17.644 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:50:17.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:50:17.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:17.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:18.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:19.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:19.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:19.132 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:50:19.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:50:19.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:19.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:19.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:20.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:20.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:20.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:50:20.671 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:50:20.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:20.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:21.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:22.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:22.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:22.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:50:22.238 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:50:22.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:22.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:22.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:23.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:23.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:23.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:50:23.838 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:50:23.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:23.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:24.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:25.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:25.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:25.453 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:50:25.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:50:25.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:25.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:25.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:26.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:26.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:26.891 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:50:26.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:50:26.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:26.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:28.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:29.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:29.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:29.084 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:50:29.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:50:29.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:29.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:29.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:30.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:30.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:30.933 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:50:30.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:50:30.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:30.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:31.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:32.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:32.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:32.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:50:32.763 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:50:32.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:32.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:33.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:34.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:34.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:34.735 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:50:34.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:50:34.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:34.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:35.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:36.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:36.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:36.754 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:50:36.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:50:36.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:36.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:37.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:38.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:38.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:38.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:50:38.363 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:50:38.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:38.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:38.865 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/400444314,v1:127.0.0.1:6819/400444314] [v2:127.0.0.1:6820/400444314,v1:127.0.0.1:6821/400444314] exists,up 7669964f-875d-4c3a-9f9c-1534a6ac4bf4 2024-04-25T07:50:38.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:50:38.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:50:38.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:50:38.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-04-25T07:50:38.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:50:39.306 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' does not exist 2024-04-25T07:50:39.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:50:39.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:50:39.876 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' already exists 2024-04-25T07:50:39.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:50:40.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:50:43.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-04-25T07:50:43.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-04-25T07:50:43.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-04-25T07:50:43.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:50:43.940 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:50:43.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-04-25T07:50:44.100 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:50:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-04-25T07:50:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-04-25T07:50:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-04-25T07:50:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:50:44.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-04-25T07:50:44.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:50:44.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:50:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:50:44.687 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:50:44.687 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:50:44.687 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:50:44.687 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:50:44.687 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:50:44.687 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:50:44.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:50:44.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:50:44.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:50:45.138 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:50:45.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:50:45.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:50:45.588 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:50:45.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:50:55.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:50:55.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:50:55.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:50:56.176 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:50:56.176 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:50:56.176 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:50:56.176 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:50:56.176 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:50:56.176 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:50:56.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:50:56.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:50:56.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:50:56.634 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:50:56.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:50:56.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:50:57.079 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:50:57.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:51:07.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:51:07.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:51:07.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:51:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:51:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:51:07.670 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:51:07.670 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:51:07.670 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:51:07.670 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:51:07.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:51:07.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:51:07.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:51:08.118 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:51:08.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:51:08.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:51:08.533 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:51:08.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:51:18.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-04-25T07:51:33.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:51:33.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-04-25T07:51:33.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-04-25T07:51:33.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-04-25T07:51:34.148 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:51:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:51:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:51:34.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:51:34.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:51:34.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:34.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:34.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:34.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:34.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:34.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:34.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:51:34.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:51:34.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:51:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:51:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:51:34.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:51:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:34.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:51:34.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:51:34.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:34.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:51:34.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:51:34.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:51:34.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:51:34.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:34.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:34.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:51:34.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:34.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:34.282 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:51:34.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:51:34.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:51:34.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:51:34.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:51:34.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:51:34.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:34.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:51:34.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:51:34.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:34.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:51:34.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:51:34.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:51:34.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:51:34.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:34.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:34.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:51:34.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:51:34.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:51:34.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:51:34.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:51:34.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:34.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:34.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176567 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:51:34.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:51:34.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:51:34.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:34.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:34.713 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:34.713 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:34.713 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:34.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:51:34.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:51:34.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:51:34.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:51:34.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:51:34.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:51:34.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:51:34.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:51:34.950 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:51:34.950 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:51:34.951 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:51:34.951 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:51:34.951 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:34.951 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:34.952 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:51:34.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:51:34.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get fsid 2024-04-25T07:51:35.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:51:35.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:51:35.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:51:35.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:51:35.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:51:35.072 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:51:35.072 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:51:35.072 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:51:35.073 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:51:35.073 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:35.073 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:35.073 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:51:35.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:51:35.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get mon_host 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:51:35.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:51:36.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:51:36.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:36.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:36.124 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:36.125 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:36.125 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:36.125 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:51:36.125 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:51:36.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:51:36.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-04-25T07:51:36.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:36.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:36.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a30d7c17-6e54-4e72-8d59-31d274eebc5e 2024-04-25T07:51:36.164 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 a30d7c17-6e54-4e72-8d59-31d274eebc5e 2024-04-25T07:51:36.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a30d7c17-6e54-4e72-8d59-31d274eebc5e' 2024-04-25T07:51:36.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:36.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCICypmTJe0ChAAV4KAHv+wwN+v1sh9MmDpTw== 2024-04-25T07:51:36.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCICypmTJe0ChAAV4KAHv+wwN+v1sh9MmDpTw=="}' 2024-04-25T07:51:36.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a30d7c17-6e54-4e72-8d59-31d274eebc5e -i td/osd-markdown/0/new.json 2024-04-25T07:51:36.535 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:51:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:51:36.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCICypmTJe0ChAAV4KAHv+wwN+v1sh9MmDpTw== --osd-uuid a30d7c17-6e54-4e72-8d59-31d274eebc5e 2024-04-25T07:51:36.583 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.583+0000 7fc9e2b3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:36.620 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.619+0000 7fc9e2b3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:36.621 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.623+0000 7fc9e2b3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:36.622 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.623+0000 7fc9e2b3f7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:51:36.622 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.623+0000 7fc9e2b3f7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:51:36.622 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.623+0000 7fc9e2b3f7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:51:36.660 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:36.659+0000 7fc9e2b3f7c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:51:43.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:51:43.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:43.992 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:51:43.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:51:43.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:44.567 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:51:44.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:51:44.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:44.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:44.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:44.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:44.598 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:44.599+0000 7fe2cad527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:44.600 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:44.599+0000 7fe2cad527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:44.601 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:44.603+0000 7fe2cad527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:44.978 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:51:44.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:44.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:45.141 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:45.139+0000 7fe2cad527c0 -1 Falling back to public interface 2024-04-25T07:51:45.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:46.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:46.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:46.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:46.462 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:51:46.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:46.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:46.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:47.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:47.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:47.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:47.884 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:51:47.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:47.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:47.893 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:51:47.895+0000 7fe2cad527c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:51:48.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:49.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:49.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:49.510 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:51:49.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:49.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:49.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:50.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:51.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:51.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:51.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:51:51.056 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:51:51.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:51.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:51.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:52.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:52.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:52.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:51:52.478 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:51:52.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:52.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:53.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:54.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:54.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:54.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:51:54.111 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:51:54.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:54.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:54.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:55.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:55.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:55.627 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:51:55.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:51:55.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:55.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:56.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:57.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:57.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:57.206 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:51:57.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:51:57.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:57.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:57.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:58.635 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:51:58.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:51:58.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:58.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:59.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:00.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:00.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:00.273 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:52:00.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:52:00.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:00.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:00.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:01.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:01.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:01.902 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:52:01.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:52:01.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:01.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:02.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:03.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:03.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:03.579 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:52:03.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:52:03.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:03.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:04.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:05.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:05.275 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:52:05.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:52:05.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:05.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:06.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:07.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:07.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:07.307 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:52:07.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:52:07.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:07.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:08.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:09.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:09.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:09.390 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:52:09.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:52:09.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:09.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:10.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:11.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:11.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:11.291 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:52:11.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:52:11.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:11.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:12.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:13.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:13.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:13.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:52:13.282 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:52:13.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:13.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:14.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:15.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:15.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:15.487 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:52:15.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:52:15.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:15.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:15.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:16.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:16.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:16.932 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:52:16.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:52:16.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:16.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:17.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:18.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:18.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:18.548 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T07:52:18.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:52:18.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:18.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:18.969 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1803645656,v1:127.0.0.1:6803/1803645656] [v2:127.0.0.1:6804/1803645656,v1:127.0.0.1:6805/1803645656] exists,up a30d7c17-6e54-4e72-8d59-31d274eebc5e 2024-04-25T07:52:18.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:52:18.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:52:18.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:52:18.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:52:18.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:52:18.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7cdfbd2b-08d4-4513-8c80-a4b625730fae 2024-04-25T07:52:18.976 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 7cdfbd2b-08d4-4513-8c80-a4b625730fae 2024-04-25T07:52:18.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7cdfbd2b-08d4-4513-8c80-a4b625730fae' 2024-04-25T07:52:18.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:52:18.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyCypmBHEwOxAAXiNBV1X+PdN8E8rb26txeg== 2024-04-25T07:52:18.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyCypmBHEwOxAAXiNBV1X+PdN8E8rb26txeg=="}' 2024-04-25T07:52:18.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7cdfbd2b-08d4-4513-8c80-a4b625730fae -i td/osd-markdown/1/new.json 2024-04-25T07:52:19.592 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:52:19.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:52:19.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyCypmBHEwOxAAXiNBV1X+PdN8E8rb26txeg== --osd-uuid 7cdfbd2b-08d4-4513-8c80-a4b625730fae 2024-04-25T07:52:19.637 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.634+0000 7f7be284c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:19.682 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.682+0000 7f7be284c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:19.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.682+0000 7f7be284c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:19.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.682+0000 7f7be284c7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:52:19.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.682+0000 7f7be284c7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:52:19.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.682+0000 7f7be284c7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:52:19.721 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:19.718+0000 7f7be284c7c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:52:27.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:52:27.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:52:27.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:52:27.034 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:52:27.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:52:27.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:52:27.620 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:52:27.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:52:27.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:52:27.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:52:27.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:52:27.650 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:27.650+0000 7fde630d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:27.652 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:27.650+0000 7fde630d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:27.653 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:27.654+0000 7fde630d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:28.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:52:28.038 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:52:28.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:28.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:28.222 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:28.222+0000 7fde630d07c0 -1 Falling back to public interface 2024-04-25T07:52:28.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:29.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:29.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:29.516 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:52:29.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:52:29.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:29.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:29.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:30.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:30.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:30.947 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:52:30.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:52:30.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:30.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:31.137 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:52:31.138+0000 7fde630d07c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:52:31.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:32.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:32.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:32.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:52:32.403 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:52:32.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:32.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:32.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:33.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:33.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:33.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:52:33.959 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:52:33.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:33.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:34.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:35.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:35.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:35.681 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:52:35.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:52:35.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:35.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:36.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:37.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:37.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:37.173 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:52:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:52:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:37.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:38.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:38.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:38.875 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:52:38.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:52:38.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:38.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:39.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:40.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:40.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:40.388 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:52:40.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:52:40.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:40.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:40.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:41.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:41.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:41.861 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:52:41.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:52:41.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:41.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:42.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:43.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:43.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:43.529 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:52:43.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:52:43.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:43.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:44.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:45.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:45.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:45.069 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:52:45.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:52:45.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:45.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:45.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:46.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:46.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:46.587 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:52:46.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:52:46.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:46.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:47.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:48.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:48.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:48.071 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:52:48.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:52:48.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:48.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:48.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:49.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:49.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:49.721 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:52:49.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:52:49.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:49.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:50.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:51.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:51.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:51.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:52:51.273 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:52:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:51.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:52.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:53.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:53.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:53.294 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:52:53.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:52:53.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:53.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:54.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:55.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:55.192 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:52:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:52:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:56.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:57.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:57.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:57.255 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:52:57.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:52:57.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:57.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:58.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:59.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:59.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:59.223 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:52:59.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:52:59.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:59.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:00.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:01.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:01.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:01.192 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T07:53:01.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:53:01.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:01.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:02.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:03.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:03.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:03.155 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T07:53:03.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:53:03.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:03.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:03.733 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3332611657,v1:127.0.0.1:6811/3332611657] [v2:127.0.0.1:6812/3332611657,v1:127.0.0.1:6813/3332611657] exists,up 7cdfbd2b-08d4-4513-8c80-a4b625730fae 2024-04-25T07:53:03.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:03.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:53:03.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:53:03.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:53:03.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:53:03.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=10327c76-8927-42fc-bcca-e7430eb2f27c 2024-04-25T07:53:03.741 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 10327c76-8927-42fc-bcca-e7430eb2f27c 2024-04-25T07:53:03.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 10327c76-8927-42fc-bcca-e7430eb2f27c' 2024-04-25T07:53:03.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:53:03.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfCypmM8JlLRAAXJ7ctzmGxonOPSUMaDG0NQ== 2024-04-25T07:53:03.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfCypmM8JlLRAAXJ7ctzmGxonOPSUMaDG0NQ=="}' 2024-04-25T07:53:03.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 10327c76-8927-42fc-bcca-e7430eb2f27c -i td/osd-markdown/2/new.json 2024-04-25T07:53:04.275 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:53:04.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:53:04.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfCypmM8JlLRAAXJ7ctzmGxonOPSUMaDG0NQ== --osd-uuid 10327c76-8927-42fc-bcca-e7430eb2f27c 2024-04-25T07:53:04.319 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.318+0000 7f81a41d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:04.322 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.322+0000 7f81a41d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:04.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.322+0000 7f81a41d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:04.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.322+0000 7f81a41d77c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:53:04.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.322+0000 7f81a41d77c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:53:04.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.322+0000 7f81a41d77c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:53:04.398 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:04.398+0000 7f81a41d77c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:53:12.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:53:12.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:53:12.353 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:53:12.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:53:12.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:53:12.956 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:53:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:53:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:53:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:53:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:53:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:53:12.987 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:12.986+0000 7f9bf2a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:13.021 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:13.022+0000 7f9bf2a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:13.023 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:13.022+0000 7f9bf2a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:13.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:53:13.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:53:13.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:53:13.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:53:13.401 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:53:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:53:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:53:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:13.608 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:13.606+0000 7f9bf2a597c0 -1 Falling back to public interface 2024-04-25T07:53:13.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:14.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:14.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:14.820 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:53:14.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:53:14.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:14.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:15.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:16.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:16.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:16.245 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:53:16.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:53:16.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:16.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:16.389 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:53:16.390+0000 7f9bf2a597c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:53:16.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:17.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:17.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:17.842 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:53:17.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:53:17.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:17.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:18.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:19.385 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:53:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:53:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:19.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:19.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:20.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:20.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:20.960 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:53:20.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:53:20.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:20.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:21.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:22.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:22.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:22.533 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:53:22.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:53:22.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:22.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:22.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:23.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:23.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:23.997 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:53:23.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:53:23.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:23.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:24.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:25.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:25.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:25.695 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:53:25.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:53:25.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:25.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:26.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:27.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:27.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:27.163 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:53:27.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:53:27.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:27.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:27.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:28.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:28.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:28.784 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:53:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:53:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:29.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:30.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:30.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:30.246 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:53:30.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:53:30.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:30.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:30.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:31.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:31.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:31.838 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:53:31.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:53:31.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:31.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:32.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:33.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:33.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:33.512 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:53:33.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:53:33.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:33.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:33.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:34.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:34.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:34.987 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:53:34.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:53:34.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:34.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:35.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:36.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:36.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:36.967 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:53:36.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:53:36.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:36.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:37.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:38.738 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:53:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:53:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:38.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:39.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:40.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:40.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:40.527 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T07:53:40.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:53:40.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:40.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:41.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:42.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:42.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:42.489 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T07:53:42.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:53:42.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:42.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:43.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:44.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:44.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:44.125 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T07:53:44.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:53:44.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:44.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:44.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:45.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:45.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:45.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:53:45.673 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T07:53:45.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:45.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:46.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:47.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:47.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:47.546 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T07:53:47.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:53:47.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:47.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:48.062 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4241437429,v1:127.0.0.1:6819/4241437429] [v2:127.0.0.1:6820/4241437429,v1:127.0.0.1:6821/4241437429] exists,up 10327c76-8927-42fc-bcca-e7430eb2f27c 2024-04-25T07:53:48.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:48.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:48.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:48.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-04-25T07:53:48.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:53:48.556 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' does not exist 2024-04-25T07:53:48.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:53:48.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:53:49.321 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' already exists 2024-04-25T07:53:49.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:53:50.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:53:53.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-04-25T07:53:53.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-04-25T07:53:53.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-04-25T07:53:53.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:53:53.260 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:53:53.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-04-25T07:53:53.464 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:53:53.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-04-25T07:53:53.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:53:53.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-04-25T07:53:53.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:53:53.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:53:53.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:53:53.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:53:53.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:53:54.088 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:53:54.088 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:53:54.088 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:53:54.088 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:53:54.088 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:53:54.089 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:53:54.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:53:54.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:53:54.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:53:54.574 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:53:54.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:53:54.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:53:55.080 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:53:55.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:05.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:54:05.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:05.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:54:05.730 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:54:05.730 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:54:05.730 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:54:05.730 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:05.730 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:54:05.730 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:54:05.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:54:05.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:54:05.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:54:06.187 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:06.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:54:06.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:54:06.692 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:54:06.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:16.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:54:16.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:16.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:54:17.313 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:54:17.313 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:54:17.314 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:54:17.314 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:17.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:54:17.314 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:54:17.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:54:17.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:54:17.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:54:17.765 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:17.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:54:17.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:54:18.310 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:54:18.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:54:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:28.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:54:28.899 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:54:28.899 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:54:28.899 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:54:28.899 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:28.899 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:54:28.899 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:54:28.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:54:28.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:54:28.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:54:29.386 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:29.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:54:29.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:54:29.862 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:54:29.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:39.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-04-25T07:54:54.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:55.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-04-25T07:54:55.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-04-25T07:54:55.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-04-25T07:54:55.547 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:55.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:54:55.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:54:55.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:54:55.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:54:55.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:54:55.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:54:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:54:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:54:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:54:55.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:54:55.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:54:55.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:54:55.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:54:55.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:54:55.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:54:55.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:54:55.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:55.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:54:55.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:54:55.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:55.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:54:55.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:54:55.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:54:55.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:54:55.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:55.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:55.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:54:55.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:54:55.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:54:55.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:54:55.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:54:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:54:55.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:54:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:54:55.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:54:55.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:54:55.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:54:55.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:54:55.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:55.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:54:55.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:54:55.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:55.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:54:55.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:54:55.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:54:55.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:54:55.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:55.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:55.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:54:55.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:54:55.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:54:55.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:54:55.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:54:55.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:55.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:55.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176567 2024-04-25T07:54:55.694 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:54:55.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:54:55.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:54:55.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:54:55.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:54:55.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:54:56.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:54:56.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:56.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:56.147 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:56.147 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:56.147 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:56.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:54:56.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:54:56.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:54:56.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:54:56.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:54:56.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:54:56.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:54:56.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:54:56.353 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:54:56.353 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:54:56.353 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:54:56.353 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:54:56.354 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:56.354 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:56.354 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:54:56.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:54:56.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get fsid 2024-04-25T07:54:56.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:54:56.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:54:56.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:54:56.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:56.474 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:56.475 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:54:56.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:54:56.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get mon_host 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:54:56.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:57.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:54:57.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:54:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:54:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:54:57.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:54:57.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:54:57.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:54:57.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aea002fe-f3b3-48ac-a628-4a7042fd8730 2024-04-25T07:54:57.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aea002fe-f3b3-48ac-a628-4a7042fd8730' 2024-04-25T07:54:57.150 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 aea002fe-f3b3-48ac-a628-4a7042fd8730 2024-04-25T07:54:57.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:54:57.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRDCpmzE71CRAAn5R4QKDteiwki4qizF/gqQ== 2024-04-25T07:54:57.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRDCpmzE71CRAAn5R4QKDteiwki4qizF/gqQ=="}' 2024-04-25T07:54:57.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aea002fe-f3b3-48ac-a628-4a7042fd8730 -i td/osd-markdown/0/new.json 2024-04-25T07:54:57.460 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:54:57.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:54:57.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRDCpmzE71CRAAn5R4QKDteiwki4qizF/gqQ== --osd-uuid aea002fe-f3b3-48ac-a628-4a7042fd8730 2024-04-25T07:54:57.509 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.504+0000 7f5a2b4b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:57.521 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.516+0000 7f5a2b4b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:57.522 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.520+0000 7f5a2b4b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:57.523 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.520+0000 7f5a2b4b87c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:54:57.523 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.520+0000 7f5a2b4b87c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:54:57.523 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.520+0000 7f5a2b4b87c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:54:57.560 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:54:57.556+0000 7f5a2b4b87c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:55:03.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:55:03.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:55:03.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:55:03.758 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:55:03.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:55:04.338 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:55:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:55:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:55:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:55:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:55:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:55:04.361 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:04.360+0000 7f65da4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:04.368 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:04.364+0000 7f65da4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:04.370 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:04.368+0000 7f65da4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:04.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:55:04.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:55:04.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:55:04.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:55:04.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:55:04.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:04.798 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:55:04.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:55:04.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:04.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:04.940 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:04.936+0000 7f65da4e27c0 -1 Falling back to public interface 2024-04-25T07:55:05.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:06.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:06.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:06.271 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:55:06.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:55:06.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:06.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:06.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:07.443 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:07.440+0000 7f65da4e27c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:55:07.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:07.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:07.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:55:07.758 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:55:07.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:07.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:08.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:09.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:09.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:09.429 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:55:09.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:55:09.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:09.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:09.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:10.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:10.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:10.867 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:55:10.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:55:10.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:10.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:11.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:12.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:12.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:12.320 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:55:12.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:55:12.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:12.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:12.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:13.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:13.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:13.760 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:55:13.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:55:13.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:13.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:14.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:15.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:15.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:15.267 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:55:15.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:55:15.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:15.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:15.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:16.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:16.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:16.703 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:55:16.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:55:16.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:16.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:17.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:18.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:18.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:18.208 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:55:18.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:55:18.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:18.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:18.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:19.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:19.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:19.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:55:19.784 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:55:19.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:19.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:20.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:21.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:21.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:21.246 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:55:21.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:55:21.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:21.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:22.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:23.220 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:55:23.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:55:23.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:23.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:24.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:25.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:25.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:25.117 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:55:25.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:55:25.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:25.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:25.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:55:26.875 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:55:26.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:26.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:27.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:28.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:28.878 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:55:28.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:28.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:55:28.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:28.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:29.483 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2237799611,v1:127.0.0.1:6803/2237799611] [v2:127.0.0.1:6804/2237799611,v1:127.0.0.1:6805/2237799611] exists,up aea002fe-f3b3-48ac-a628-4a7042fd8730 2024-04-25T07:55:29.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:55:29.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:55:29.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:55:29.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:55:29.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:55:29.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:55:29.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:55:29.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:55:29.486 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:55:29.486 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:55:29.486 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:55:29.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:55:29.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:55:29.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:55:29.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:55:29.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:55:29.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:55:29.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d95c1811-3966-44fa-bb8f-613f82d2d597 2024-04-25T07:55:29.489 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 d95c1811-3966-44fa-bb8f-613f82d2d597 2024-04-25T07:55:29.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d95c1811-3966-44fa-bb8f-613f82d2d597' 2024-04-25T07:55:29.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:55:29.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxDCpm/LE4HhAA/XDBTzv3G5wwmYEgKa6wZA== 2024-04-25T07:55:29.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxDCpm/LE4HhAA/XDBTzv3G5wwmYEgKa6wZA=="}' 2024-04-25T07:55:29.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d95c1811-3966-44fa-bb8f-613f82d2d597 -i td/osd-markdown/1/new.json 2024-04-25T07:55:30.075 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:55:30.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:55:30.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxDCpm/LE4HhAA/XDBTzv3G5wwmYEgKa6wZA== --osd-uuid d95c1811-3966-44fa-bb8f-613f82d2d597 2024-04-25T07:55:30.121 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.120+0000 7f420cb827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:30.124 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.124+0000 7f420cb827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:30.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.128+0000 7f420cb827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:30.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.128+0000 7f420cb827c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:55:30.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.128+0000 7f420cb827c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:55:30.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.128+0000 7f420cb827c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:55:30.163 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:30.164+0000 7f420cb827c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:55:35.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:55:35.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:55:35.940 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:55:35.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:55:35.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:55:36.516 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:55:36.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:55:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:55:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:55:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:55:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:55:36.547 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:36.547+0000 7fe72bcb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:36.549 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:36.551+0000 7fe72bcb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:36.550 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:36.551+0000 7fe72bcb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:36.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:55:36.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:37.106 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:37.107+0000 7fe72bcb57c0 -1 Falling back to public interface 2024-04-25T07:55:37.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:38.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:38.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:38.446 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:55:38.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:55:38.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:38.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:38.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:39.543 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:55:39.543+0000 7fe72bcb57c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:55:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:39.913 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:55:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:55:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:40.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:41.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:41.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:41.575 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:55:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:55:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:42.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:43.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:43.041 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:55:43.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:43.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:55:43.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:43.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:44.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:44.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:44.600 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:55:44.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:55:44.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:44.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:45.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:46.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:46.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:46.099 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:55:46.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:55:46.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:46.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:46.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:47.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:47.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:47.522 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:55:47.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:55:47.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:47.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:47.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:48.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:48.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:48.953 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:55:48.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:55:48.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:48.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:49.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:50.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:50.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:50.543 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:55:50.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:55:50.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:50.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:51.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:52.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:52.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:52.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:55:52.050 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:55:52.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:52.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:52.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:53.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:53.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:53.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:55:53.820 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:55:53.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:53.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:54.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:55.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:55.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:55.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:55:55.668 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:55:55.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:55.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:56.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:57.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:57.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:57.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:55:57.403 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:55:57.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:57.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:59.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:59.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:59.103 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:55:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:55:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:59.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:00.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:00.936 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:56:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:56:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:56:01.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:02.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:02.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:02.500 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:56:02.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:56:02.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:02.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:56:03.048 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1797825967,v1:127.0.0.1:6811/1797825967] [v2:127.0.0.1:6812/1797825967,v1:127.0.0.1:6813/1797825967] exists,up d95c1811-3966-44fa-bb8f-613f82d2d597 2024-04-25T07:56:03.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:56:03.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:56:03.053 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:56:03.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:56:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:56:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:56:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:56:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:56:03.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:56:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf9e3e45-a4f6-427f-b5f0-69730dfcfc9c 2024-04-25T07:56:03.056 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 bf9e3e45-a4f6-427f-b5f0-69730dfcfc9c 2024-04-25T07:56:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bf9e3e45-a4f6-427f-b5f0-69730dfcfc9c' 2024-04-25T07:56:03.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:56:03.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTDCpm1nyQBBAAZCsfUsx9+sfW5DgNi/bXgw== 2024-04-25T07:56:03.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTDCpm1nyQBBAAZCsfUsx9+sfW5DgNi/bXgw=="}' 2024-04-25T07:56:03.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf9e3e45-a4f6-427f-b5f0-69730dfcfc9c -i td/osd-markdown/2/new.json 2024-04-25T07:56:03.693 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:56:03.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:56:03.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTDCpm1nyQBBAAZCsfUsx9+sfW5DgNi/bXgw== --osd-uuid bf9e3e45-a4f6-427f-b5f0-69730dfcfc9c 2024-04-25T07:56:03.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.755+0000 7fac20efa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:03.756 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.755+0000 7fac20efa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:03.757 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.759+0000 7fac20efa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:03.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.759+0000 7fac20efa7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:56:03.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.759+0000 7fac20efa7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:56:03.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.759+0000 7fac20efa7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:56:03.849 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:03.851+0000 7fac20efa7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:56:09.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:56:09.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:56:09.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:56:09.877 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:56:09.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:56:10.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:56:10.440 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:56:10.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:56:10.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:56:10.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:56:10.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:56:10.469 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:10.471+0000 7f6d48fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:10.482 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:10.483+0000 7f6d48fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:10.483 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:10.483+0000 7f6d48fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:10.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:56:10.870 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:56:10.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:10.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:11.039 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:11.039+0000 7f6d48fb47c0 -1 Falling back to public interface 2024-04-25T07:56:11.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:12.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:12.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:12.337 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:56:12.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:56:12.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:12.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:12.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:13.649 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:56:13.647+0000 7f6d48fb47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:56:13.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:13.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:13.814 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:56:13.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:56:13.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:13.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:14.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:15.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:15.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:15.284 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:56:15.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:56:15.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:15.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:15.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:16.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:16.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:16.760 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:56:16.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:56:16.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:16.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:17.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:18.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:18.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:18.198 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:56:18.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:56:18.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:18.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:18.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:19.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:19.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:19.651 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:56:19.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:56:19.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:19.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:20.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:21.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:56:21.088 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:56:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:21.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:22.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:22.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:22.632 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:56:22.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:56:22.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:22.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:24.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:24.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:24.053 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:56:24.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:56:24.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:24.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:24.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:25.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:25.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:25.616 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:56:25.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:56:25.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:25.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:26.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:27.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:27.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:27.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:56:27.335 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:56:27.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:27.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:27.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:28.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:28.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:28.776 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:56:28.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:56:28.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:28.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:29.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:30.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:30.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:30.515 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:56:30.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:56:30.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:30.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:31.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:32.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:32.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:32.209 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:56:32.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:56:32.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:32.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:32.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:33.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:33.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:33.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:56:33.920 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:56:33.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:33.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:34.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:56:35.632 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:56:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:35.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:36.162 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3795246686,v1:127.0.0.1:6819/3795246686] [v2:127.0.0.1:6820/3795246686,v1:127.0.0.1:6821/3795246686] exists,up bf9e3e45-a4f6-427f-b5f0-69730dfcfc9c 2024-04-25T07:56:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:56:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:56:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:56:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-04-25T07:56:36.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:56:36.635 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' does not exist 2024-04-25T07:56:36.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:56:36.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:56:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' already exists 2024-04-25T07:56:37.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:56:38.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:56:39.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-04-25T07:56:39.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-04-25T07:56:39.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-04-25T07:56:39.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:56:39.779 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:56:39.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-04-25T07:56:39.935 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T07:56:39.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-04-25T07:56:39.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:56:39.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-04-25T07:56:39.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:56:39.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:56:39.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:39.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:40.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:40.664 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:40.664 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:56:40.664 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:56:40.664 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:40.664 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:40.664 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:40.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:40.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:40.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:41.129 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:41.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:41.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:41.537 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:56:41.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:56:51.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:51.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:51.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:52.159 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:52.159 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:56:52.159 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:56:52.160 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:52.160 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:52.160 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:52.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:52.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:52.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:52.646 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:52.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:52.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:53.110 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:56:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:57:03.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:57:03.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:57:03.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:57:03.747 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:57:03.747 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:57:03.747 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:57:03.747 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:03.747 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:57:03.747 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:57:03.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:57:03.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:57:03.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:57:04.232 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:04.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:57:04.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:57:04.679 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:57:04.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:57:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:57:14.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:57:14.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:57:15.311 INFO:tasks.workunit.client.0.smithi031.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:57:15.311 INFO:tasks.workunit.client.0.smithi031.stdout:-1 0.29306 root default 2024-04-25T07:57:15.311 INFO:tasks.workunit.client.0.smithi031.stdout:-3 0.29306 host smithi031 2024-04-25T07:57:15.311 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:15.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:57:15.311 INFO:tasks.workunit.client.0.smithi031.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:57:15.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:57:15.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:57:15.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:57:15.797 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:15.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:57:15.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:57:16.286 INFO:tasks.workunit.client.0.smithi031.stderr:marked down osd.0. 2024-04-25T07:57:16.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:57:26.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-04-25T07:57:26.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-04-25T07:57:26.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-04-25T07:57:26.739 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T07:57:26.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:57:26.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:57:26.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:57:26.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:57:26.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:57:26.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:57:26.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:57:26.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:57:26.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:57:26.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:57:26.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:57:26.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:57:26.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:57:26.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:57:26.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:57:26.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:57:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:57:26.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:57:26.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:26.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:57:26.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:57:26.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:57:26.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:57:26.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:26.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:26.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:57:26.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:57:26.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:57:26.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:57:26.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:57:26.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:57:26.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:57:26.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:57:26.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:57:26.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:57:26.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:57:26.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:57:26.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:57:26.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:57:26.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:26.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:57:26.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:57:26.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:26.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:57:26.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:57:26.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:57:26.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:57:26.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:26.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:26.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:57:26.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:57:26.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:57:26.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:57:26.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:57:26.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:26.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:26.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176567 2024-04-25T07:57:26.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:57:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:57:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:57:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:57:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-04-25T07:57:26.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:57:26.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:57:27.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:57:27.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:27.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:27.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:27.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:27.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:27.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:57:27.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:57:27.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:57:27.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:57:27.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:57:27.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:57:27.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:57:27.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:57:27.525 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:57:27.526 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:57:27.526 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:57:27.526 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:57:27.526 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:27.527 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:27.527 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:57:27.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:57:27.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get fsid 2024-04-25T07:57:27.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:57:27.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:57:27.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:57:27.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:57:27.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:57:27.646 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:57:27.646 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:57:27.646 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:57:27.647 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:57:27.647 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:27.647 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:27.647 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176567/ceph-mon.a.asok 2024-04-25T07:57:27.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:57:27.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176567/ceph-mon.a.asok config get mon_host 2024-04-25T07:57:27.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-04-25T07:57:27.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:57:27.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:57:27.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:57:27.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:57:27.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:57:27.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:57:28.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:57:28.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:28.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:28.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:28.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:28.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:28.667 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:57:28.667 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:57:28.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:57:28.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:28.732 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:57:28.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:57:28.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:57:28.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:57:28.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:57:28.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=895ec94e-1210-4336-a34a-a0433b804135 2024-04-25T07:57:28.735 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 895ec94e-1210-4336-a34a-a0433b804135 2024-04-25T07:57:28.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 895ec94e-1210-4336-a34a-a0433b804135' 2024-04-25T07:57:28.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:57:28.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoDCpmzNjeLBAA1isbzeRRSCNfg0hC9oEPYw== 2024-04-25T07:57:28.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoDCpmzNjeLBAA1isbzeRRSCNfg0hC9oEPYw=="}' 2024-04-25T07:57:28.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 895ec94e-1210-4336-a34a-a0433b804135 -i td/osd-markdown/0/new.json 2024-04-25T07:57:29.026 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:57:29.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:57:29.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoDCpmzNjeLBAA1isbzeRRSCNfg0hC9oEPYw== --osd-uuid 895ec94e-1210-4336-a34a-a0433b804135 2024-04-25T07:57:29.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.082+0000 7fc58dd5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:29.086 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.086+0000 7fc58dd5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:29.088 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.086+0000 7fc58dd5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:29.088 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.086+0000 7fc58dd5b7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:29.089 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.086+0000 7fc58dd5b7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:29.089 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.086+0000 7fc58dd5b7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:29.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:29.126+0000 7fc58dd5b7c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:57:34.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:57:34.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:57:34.849 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:57:34.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:57:34.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:57:35.431 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:57:35.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:57:35.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:57:35.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:57:35.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:57:35.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:57:35.462 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:35.462+0000 7f30868617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:35.465 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:35.466+0000 7f30868617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:35.466 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:35.466+0000 7f30868617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:35.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:57:35.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:57:35.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:57:35.885 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:57:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:57:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:57:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:57:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:36.058 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:36.058+0000 7f30868617c0 -1 Falling back to public interface 2024-04-25T07:57:36.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:37.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:37.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:37.353 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:57:37.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:57:37.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:37.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:37.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:38.600 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:57:38.598+0000 7f30868617c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:57:38.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:38.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:38.817 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:57:38.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:57:38.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:38.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:39.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:40.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:40.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:40.336 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:57:40.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:57:40.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:40.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:40.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:41.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:41.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:41.770 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:57:41.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:57:41.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:41.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:42.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:43.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:43.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:43.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:57:43.346 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:57:43.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:43.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:43.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:44.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:44.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:44.786 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:57:44.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:57:44.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:44.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:45.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:46.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:46.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:46.332 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:57:46.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:57:46.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:46.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:46.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:47.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:47.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:47.806 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:57:47.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:57:47.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:47.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:48.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:49.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:49.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:49.369 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:57:49.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:57:49.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:49.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:49.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:50.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:50.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:50.862 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:57:50.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:57:50.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:50.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:51.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:52.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:52.636 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:57:52.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:52.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:57:52.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:52.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:54.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:54.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:54.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:57:54.318 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:57:54.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:54.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:54.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:55.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:55.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:55.943 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:57:55.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:57:55.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:55.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:57.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:58.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:58.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:58.099 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:57:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:57:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:59.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:00.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:00.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:00.013 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:58:00.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:58:00.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:00.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:00.495 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3895012825,v1:127.0.0.1:6803/3895012825] [v2:127.0.0.1:6804/3895012825,v1:127.0.0.1:6805/3895012825] exists,up 895ec94e-1210-4336-a34a-a0433b804135 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:58:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:58:00.499 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:58:00.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:58:00.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:58:00.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:58:00.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=08ff179c-e4fe-45a5-81b2-38a56d4a7185 2024-04-25T07:58:00.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 08ff179c-e4fe-45a5-81b2-38a56d4a7185' 2024-04-25T07:58:00.501 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 08ff179c-e4fe-45a5-81b2-38a56d4a7185 2024-04-25T07:58:00.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:58:00.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIDSpmnkv/HhAAuk12zWas2SeHTBkyJy8CVQ== 2024-04-25T07:58:00.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIDSpmnkv/HhAAuk12zWas2SeHTBkyJy8CVQ=="}' 2024-04-25T07:58:00.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 08ff179c-e4fe-45a5-81b2-38a56d4a7185 -i td/osd-markdown/1/new.json 2024-04-25T07:58:01.124 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:58:01.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:58:01.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIDSpmnkv/HhAAuk12zWas2SeHTBkyJy8CVQ== --osd-uuid 08ff179c-e4fe-45a5-81b2-38a56d4a7185 2024-04-25T07:58:01.179 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.177+0000 7fc6b9b847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:01.182 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.181+0000 7fc6b9b847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:01.183 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.181+0000 7fc6b9b847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:01.183 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.181+0000 7fc6b9b847c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:58:01.184 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.181+0000 7fc6b9b847c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:58:01.184 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.181+0000 7fc6b9b847c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:58:01.222 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:01.221+0000 7fc6b9b847c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:58:07.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:58:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:58:07.063 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T07:58:07.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:58:07.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:58:07.635 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T07:58:07.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:58:07.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:58:07.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:58:07.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:58:07.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:58:07.665 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:07.665+0000 7fee37d5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:07.677+0000 7fee37d5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:07.677+0000 7fee37d5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:08.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:58:08.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:58:08.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:58:08.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:58:08.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:58:08.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:08.070 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:58:08.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:58:08.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:08.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:08.246 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:08.245+0000 7fee37d5b7c0 -1 Falling back to public interface 2024-04-25T07:58:08.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:09.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:09.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:09.515 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:58:09.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:58:09.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:09.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:09.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:10.815 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:10.813+0000 7fee37d5b7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:58:10.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:10.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:10.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:58:10.943 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:58:10.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:10.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:11.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:12.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:12.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:12.569 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:58:12.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:58:12.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:12.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:13.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:14.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:14.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:14.005 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:58:14.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:58:14.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:14.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:14.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:15.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:15.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:15.435 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:58:15.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:58:15.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:15.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:15.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:16.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:16.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:16.896 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:58:16.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:58:16.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:16.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:17.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:18.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:18.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:18.333 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:58:18.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:58:18.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:18.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:18.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:19.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:19.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:19.891 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:58:19.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:58:19.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:19.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:21.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:21.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:21.380 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:58:21.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:58:21.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:21.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:22.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:23.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:23.147 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:58:23.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:23.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:58:23.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:23.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:23.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:24.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:24.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:24.698 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:58:24.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:58:24.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:24.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:25.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:26.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:26.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:26.350 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:58:26.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:58:26.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:26.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:27.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:28.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:28.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:28.463 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:58:28.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:58:28.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:28.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:29.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:30.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:30.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:30.420 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:58:30.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:58:30.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:30.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:31.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:32.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:32.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:32.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:58:32.168 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:58:32.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:32.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:32.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:33.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:33.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:33.863 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:58:33.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:58:33.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:33.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:34.337 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2754221695,v1:127.0.0.1:6811/2754221695] [v2:127.0.0.1:6812/2754221695,v1:127.0.0.1:6813/2754221695] exists,up 08ff179c-e4fe-45a5-81b2-38a56d4a7185 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:58:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:58:34.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:58:34.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1bf012e-4eff-46d0-9482-b61ad756b22b 2024-04-25T07:58:34.344 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 d1bf012e-4eff-46d0-9482-b61ad756b22b 2024-04-25T07:58:34.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d1bf012e-4eff-46d0-9482-b61ad756b22b' 2024-04-25T07:58:34.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:58:34.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqDSpmptbDFRAAQYNBKTMefCSwkkqBQtn+gQ== 2024-04-25T07:58:34.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqDSpmptbDFRAAQYNBKTMefCSwkkqBQtn+gQ=="}' 2024-04-25T07:58:34.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1bf012e-4eff-46d0-9482-b61ad756b22b -i td/osd-markdown/2/new.json 2024-04-25T07:58:34.854 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:58:34.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:58:34.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqDSpmptbDFRAAQYNBKTMefCSwkkqBQtn+gQ== --osd-uuid d1bf012e-4eff-46d0-9482-b61ad756b22b 2024-04-25T07:58:34.903 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.901+0000 7f648f33d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:34.905 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.905+0000 7f648f33d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:34.907 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.905+0000 7f648f33d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:34.907 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.905+0000 7f648f33d7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:58:34.907 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.905+0000 7f648f33d7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:58:34.907 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.905+0000 7f648f33d7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:58:34.950 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:34.949+0000 7f648f33d7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:58:41.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:58:41.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:58:41.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:58:41.062 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T07:58:41.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:58:41.637 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T07:58:41.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:58:41.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:58:41.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:58:41.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=dfca9df3-649a-470c-a6d6-57f25b4d86cf --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176567/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:58:41.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:58:41.670 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:41.669+0000 7f6f2879f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:41.675 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:41.673+0000 7f6f2879f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:41.676 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:41.673+0000 7f6f2879f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:42.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:58:42.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:58:42.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:42.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:42.230 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:42.229+0000 7f6f2879f7c0 -1 Falling back to public interface 2024-04-25T07:58:42.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:43.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:43.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:43.543 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:58:43.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:58:43.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:43.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:43.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:44.734 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:58:44.733+0000 7f6f2879f7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:58:44.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:44.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:44.968 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:58:44.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:58:44.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:44.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:45.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:46.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:46.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:46.562 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:58:46.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:58:46.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:46.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:46.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:47.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:47.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:47.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:58:47.986 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:58:47.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:47.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:48.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:49.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:49.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:49.564 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:58:49.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:58:49.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:49.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:50.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:51.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:51.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:51.084 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:58:51.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:58:51.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:51.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:51.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:52.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:52.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:52.562 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:58:52.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:58:52.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:52.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:53.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:54.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:54.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:54.022 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:58:54.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:58:54.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:54.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:54.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:55.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:55.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:55.567 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:58:55.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:58:55.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:55.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:56.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:57.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:57.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:57.096 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:58:57.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:58:57.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:57.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:57.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:58.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:58.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:58.529 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:58:58.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:58:58.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:58.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:59.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:00.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:00.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:00.180 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:59:00.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:59:00.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:00.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:01.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:02.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:02.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:02.223 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:59:02.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:59:02.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:02.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:02.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:03.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:03.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:03.741 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:59:03.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:59:03.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:03.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:04.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:05.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:05.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:05.520 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:59:05.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:59:05.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:05.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:06.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:07.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:07.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:07.501 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:59:07.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:59:07.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:07.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:07.955 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1209349044,v1:127.0.0.1:6819/1209349044] [v2:127.0.0.1:6820/1209349044,v1:127.0.0.1:6821/1209349044] exists,up d1bf012e-4eff-46d0-9482-b61ad756b22b 2024-04-25T07:59:07.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:59:07.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:59:07.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:59:07.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-04-25T07:59:07.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=195873 2024-04-25T07:59:07.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 195873 2024-04-25T07:59:07.964 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2024-04-25T07:59:07.964 INFO:tasks.workunit.client.0.smithi031.stdout: 195873 ? 00:00:00 ceph-osd 2024-04-25T07:59:07.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-04-25T07:59:07.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-04-25T07:59:07.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-04-25T07:59:08.393 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:08.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-04-25T07:59:08.933 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already stopped. 2024-04-25T07:59:08.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-04-25T07:59:23.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-04-25T07:59:23.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-04-25T07:59:23.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-04-25T07:59:24.415 INFO:tasks.workunit.client.0.smithi031.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T07:59:24.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 195873 2024-04-25T07:59:24.425 INFO:tasks.workunit.client.0.smithi031.stdout: PID TTY TIME CMD 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:59:24.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:59:24.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:59:24.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:59:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:59:24.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:59:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:59:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:59:24.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:59:24.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:59:24.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:59:24.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:59:24.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:59:24.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:59:24.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:59:24.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:59:24.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:59:24.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:24.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:59:24.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:59:24.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:59:24.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-markdown 0 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:59:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:59:24.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:59:24.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:59:24.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:59:24.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:59:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:59:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:59:24.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:59:24.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:59:24.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:59:24.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:59:24.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:59:24.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:59:24.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:59:24.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:59:24.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:59:24.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:24.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176567 2024-04-25T07:59:24.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176567 2024-04-25T07:59:24.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:59:24.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:59:24.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:59:24.563 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:59:24.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:59:24.573 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-04-25T07:59:24.573 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-04-25T07:59:24.626 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:59:24.631 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:59:24.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:59:24.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:59:24.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reactivate 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-04-25T07:59:24.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-04-25T07:59:24.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=cdb4afaa-348e-47d9-89db-315fdf1454b0 --auth-supported=none ' 2024-04-25T07:59:24.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-04-25T07:59:24.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-04-25T07:59:24.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:59:24.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-04-25T07:59:24.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:59:24.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:59:24.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:59:24.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:59:24.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:59:24.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:59:24.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:59:24.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:59:24.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:59:24.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:59:24.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:59:24.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:59:24.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:59:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:59:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:59:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T07:59:24.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:59:24.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:24.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:24.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.200284 2024-04-25T07:59:24.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:59:24.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:59:24.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-04-25T07:59:24.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:59:24.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:24.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:24.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.200284 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:59:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:59:24.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:59:24.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-04-25T07:59:24.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-04-25T07:59:25.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:59:25.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:59:25.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:59:25.107 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:59:25.108 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:25.108 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:25.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200284/$cluster-$name.asok' 2024-04-25T07:59:25.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:59:25.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:59:25.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:59:25.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:59:25.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:59:25.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:59:25.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:59:25.320 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.200284/ceph-mon.a.asok 2024-04-25T07:59:25.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:59:25.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.200284/ceph-mon.a.asok config get fsid 2024-04-25T07:59:25.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:59:25.440 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:59:25.440 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:25.440 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:25.440 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.200284/ceph-mon.a.asok 2024-04-25T07:59:25.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:59:25.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.200284/ceph-mon.a.asok config get mon_host 2024-04-25T07:59:25.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-04-25T07:59:25.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-04-25T07:59:25.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:59:25.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:59:25.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:59:25.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-04-25T07:59:25.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:59:26.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:59:26.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:59:26.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:59:26.450 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:59:26.450 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:26.450 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:26.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200284/$cluster-$name.asok' 2024-04-25T07:59:26.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:59:26.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cdb4afaa-348e-47d9-89db-315fdf1454b0 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:59:26.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T07:59:26.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200284/$cluster-$name.asok' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:59:26.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-04-25T07:59:26.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:59:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd9c28a7-11ea-466d-a80b-1bd8822e43ba 2024-04-25T07:59:26.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cd9c28a7-11ea-466d-a80b-1bd8822e43ba' 2024-04-25T07:59:26.491 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 cd9c28a7-11ea-466d-a80b-1bd8822e43ba 2024-04-25T07:59:26.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:59:26.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeDSpmly9GHhAAJUSJktGGSS8d4rYcHSNEkA== 2024-04-25T07:59:26.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeDSpmly9GHhAAJUSJktGGSS8d4rYcHSNEkA=="}' 2024-04-25T07:59:26.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd9c28a7-11ea-466d-a80b-1bd8822e43ba -i td/osd-reactivate/0/new.json 2024-04-25T07:59:26.813 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:59:26.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-04-25T07:59:26.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cdb4afaa-348e-47d9-89db-315fdf1454b0 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeDSpmly9GHhAAJUSJktGGSS8d4rYcHSNEkA== --osd-uuid cd9c28a7-11ea-466d-a80b-1bd8822e43ba 2024-04-25T07:59:26.861 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.860+0000 7f569a52a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:26.882 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.880+0000 7f569a52a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:26.883 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.880+0000 7f569a52a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:26.883 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.880+0000 7f569a52a7c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T07:59:26.883 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.880+0000 7f569a52a7c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T07:59:26.883 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.880+0000 7f569a52a7c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T07:59:26.921 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:26.920+0000 7f569a52a7c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-04-25T07:59:33.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-04-25T07:59:33.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:59:33.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:59:33.160 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T07:59:33.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:59:33.732 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T07:59:33.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:59:33.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cdb4afaa-348e-47d9-89db-315fdf1454b0 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:59:33.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:59:33.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:59:33.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:59:33.762 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:33.760+0000 7f277acf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:33.764 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:33.760+0000 7f277acf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:33.766 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:33.764+0000 7f277acf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:34.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:59:34.194 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T07:59:34.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:34.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:34.327 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:34.324+0000 7f277acf47c0 -1 Falling back to public interface 2024-04-25T07:59:34.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:35.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:35.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:35.644 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T07:59:35.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:59:35.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:35.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:36.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:36.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T07:59:36.800+0000 7f277acf47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:59:37.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:37.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:37.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:59:37.123 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T07:59:37.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:37.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:37.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:38.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:38.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:38.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:59:38.709 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T07:59:38.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:38.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:39.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:40.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:40.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:40.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:59:40.165 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T07:59:40.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:40.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:40.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:41.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:41.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:41.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:59:41.664 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T07:59:41.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:41.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:42.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:43.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:43.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:43.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:59:43.094 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T07:59:43.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:43.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:43.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:44.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:44.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:44.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:59:44.531 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T07:59:44.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:44.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:44.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:45.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:45.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:45.960 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T07:59:45.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:59:45.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:45.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:46.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:47.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:47.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:47.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:59:47.545 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T07:59:47.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:47.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:48.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:49.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:49.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:49.192 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T07:59:49.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:59:49.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:49.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:49.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:50.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:50.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:50.656 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T07:59:50.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:59:50.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:50.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:51.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:52.106 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T07:59:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:59:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:53.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:53.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:53.773 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T07:59:53.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:59:53.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:53.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:54.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:55.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:55.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:55.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:59:55.790 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T07:59:55.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:55.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:56.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:57.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:57.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:57.599 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T07:59:57.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:59:57.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:57.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:59:58.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:59.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:59.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:59.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:59:59.561 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T07:59:59.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:59.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:00.022 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2316536138,v1:127.0.0.1:6803/2316536138] [v2:127.0.0.1:6804/2316536138,v1:127.0.0.1:6805/2316536138] exists,up cd9c28a7-11ea-466d-a80b-1bd8822e43ba 2024-04-25T08:00:00.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:00:00.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:00:00.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:00:00.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-04-25T08:00:00.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:00.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:00.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:00.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:00.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:00.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:00.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-04-25T08:00:00.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-04-25T08:00:00.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-04-25T08:00:00.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-04-25T08:00:00.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-04-25T08:00:00.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:00:00.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:00:00.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:00:00.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cdb4afaa-348e-47d9-89db-315fdf1454b0 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:00.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.200284/$cluster-$name.asok' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:00:00.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:00:00.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:00:00.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:00:00.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-04-25T08:00:00.339 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:00:00.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:00:00.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=cdb4afaa-348e-47d9-89db-315fdf1454b0 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.200284/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:00:00.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-04-25T08:00:00.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:00:00.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:00:00.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:00:00.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:00:00.369 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:00.372+0000 7f277028d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:00.376 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:00.376+0000 7f277028d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:00.377 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:00.380+0000 7f277028d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:00:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:00:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:00:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:00:00.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:00:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:00.805 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:00:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:00:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:00.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:00.954 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:00.956+0000 7f277028d7c0 -1 Falling back to public interface 2024-04-25T08:00:01.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:02.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:02.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:02.251 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:00:02.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:00:02.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:02.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:02.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:03.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:03.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:03.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:00:03.724 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:00:03.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:03.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:03.942 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:03.944+0000 7f277028d7c0 -1 osd.0 6 log_to_monitors true 2024-04-25T08:00:04.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:05.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:05.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:05.189 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:00:05.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:00:05.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:05.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:05.616 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/3714877101,v1:127.0.0.1:6803/3714877101] [v2:127.0.0.1:6804/3714877101,v1:127.0.0.1:6805/3714877101] exists,up cd9c28a7-11ea-466d-a80b-1bd8822e43ba 2024-04-25T08:00:05.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:00:05.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:05.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:05.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:05.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:05.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:05.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:05.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:05.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:05.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:05.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:05.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:05.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:05.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:05.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:05.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:00:05.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:00:05.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:05.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:05.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T08:00:05.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.200284 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reactivate 0 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:05.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:05.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:05.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:05.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:05.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:05.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:05.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:05.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:05.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:05.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:05.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:05.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:05.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:05.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:00:05.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:00:05.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:05.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:05.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.200284 2024-04-25T08:00:05.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.200284 2024-04-25T08:00:05.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:05.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:05.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:00:05.747 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:00:05.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:00:05.798 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-04-25T08:00:05.798 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-04-25T08:00:05.851 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:00:05.856 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:00:05.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:00:05.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-prio 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-04-25T08:00:05.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none ' 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-04-25T08:00:05.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_priority' 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:05.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:05.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:05.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:05.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:05.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:05.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:05.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:05.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:05.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:05.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:05.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:05.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:05.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:05.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:05.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:00:05.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202871 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:05.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T08:00:05.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:00:05.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:05.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:05.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.202871 2024-04-25T08:00:05.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:00:05.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:00:05.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T08:00:05.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T08:00:06.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:00:06.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:06.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:06.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:00:06.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:00:06.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:00:06.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:00:06.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:00:06.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:00:06.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:00:06.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:00:06.527 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:00:06.527 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:00:06.527 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:00:06.527 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:00:06.527 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:06.527 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:06.528 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-mon.a.asok 2024-04-25T08:00:06.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:00:06.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202871/ceph-mon.a.asok config get fsid 2024-04-25T08:00:06.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:00:06.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:00:06.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:00:06.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:00:06.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:00:06.644 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:00:06.644 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:00:06.644 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:00:06.644 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:00:06.644 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:06.644 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:06.645 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-mon.a.asok 2024-04-25T08:00:06.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:00:06.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202871/ceph-mon.a.asok config get mon_host 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T08:00:06.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:07.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:00:07.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:00:07.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:00:07.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-04-25T08:00:07.671 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-04-25T08:00:07.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T08:00:07.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:07.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T08:00:07.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:00:07.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e1f79f5-1fad-450c-a44b-f00d6d7ce0f3 2024-04-25T08:00:07.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6e1f79f5-1fad-450c-a44b-f00d6d7ce0f3' 2024-04-25T08:00:07.679 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 6e1f79f5-1fad-450c-a44b-f00d6d7ce0f3 2024-04-25T08:00:07.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:00:07.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHDSpmOkSgKRAAl0k+7VsT69KLm+d5rnCi9g== 2024-04-25T08:00:07.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHDSpmOkSgKRAAl0k+7VsT69KLm+d5rnCi9g=="}' 2024-04-25T08:00:07.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e1f79f5-1fad-450c-a44b-f00d6d7ce0f3 -i td/osd-recovery-prio/0/new.json 2024-04-25T08:00:08.012 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:00:08.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T08:00:08.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHDSpmOkSgKRAAl0k+7VsT69KLm+d5rnCi9g== --osd-uuid 6e1f79f5-1fad-450c-a44b-f00d6d7ce0f3 2024-04-25T08:00:08.063 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.064+0000 7f5c8704d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:08.080 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.080+0000 7f5c8704d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:08.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.084+0000 7f5c8704d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:08.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.084+0000 7f5c8704d7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:00:08.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.084+0000 7f5c8704d7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:00:08.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.084+0000 7f5c8704d7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:00:08.120 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:08.120+0000 7f5c8704d7c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T08:00:13.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T08:00:13.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:00:13.933 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:00:13.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:00:13.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:00:14.510 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:00:14.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:00:14.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:00:14.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:00:14.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:00:14.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:00:14.547 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:14.547+0000 7f4ef1c7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:14.552 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:14.551+0000 7f4ef1c7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:14.553 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:14.555+0000 7f4ef1c7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:14.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:00:14.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:00:14.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:00:14.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:00:14.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:00:14.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:14.951 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:00:14.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:00:14.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:14.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:15.122 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:15.123+0000 7f4ef1c7b7c0 -1 Falling back to public interface 2024-04-25T08:00:15.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:16.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:16.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:16.381 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:00:16.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:00:16.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:16.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:16.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:17.637 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:17.639+0000 7f4ef1c7b7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:00:17.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:17.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:17.856 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:00:17.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:00:17.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:17.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:18.446 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4080490873,v1:127.0.0.1:6803/4080490873] [v2:127.0.0.1:6804/4080490873,v1:127.0.0.1:6805/4080490873] exists,up 6e1f79f5-1fad-450c-a44b-f00d6d7ce0f3 2024-04-25T08:00:18.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:00:18.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:00:18.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:00:18.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:00:18.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:00:18.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:00:18.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:00:18.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T08:00:18.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:00:18.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb1aa7b7-2c33-4d44-8889-234cd68aeca0 2024-04-25T08:00:18.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eb1aa7b7-2c33-4d44-8889-234cd68aeca0' 2024-04-25T08:00:18.453 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 eb1aa7b7-2c33-4d44-8889-234cd68aeca0 2024-04-25T08:00:18.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:00:18.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSDSpm9RqKHBAA2CHJROIkZqXPpfPSlhI7pA== 2024-04-25T08:00:18.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSDSpm9RqKHBAA2CHJROIkZqXPpfPSlhI7pA=="}' 2024-04-25T08:00:18.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb1aa7b7-2c33-4d44-8889-234cd68aeca0 -i td/osd-recovery-prio/1/new.json 2024-04-25T08:00:19.067 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:00:19.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T08:00:19.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSDSpm9RqKHBAA2CHJROIkZqXPpfPSlhI7pA== --osd-uuid eb1aa7b7-2c33-4d44-8889-234cd68aeca0 2024-04-25T08:00:19.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.127+0000 7ff7680d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:19.129 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.131+0000 7ff7680d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:19.130 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.131+0000 7ff7680d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:19.130 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.131+0000 7ff7680d77c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:00:19.131 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.131+0000 7ff7680d77c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:00:19.131 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.131+0000 7ff7680d77c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:00:19.217 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:19.219+0000 7ff7680d77c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T08:00:24.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T08:00:24.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:00:24.754 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:00:24.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:00:24.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:00:25.335 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:00:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:00:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:00:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:00:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:00:25.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:00:25.371 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:25.371+0000 7faffe8877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:25.379 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:25.379+0000 7faffe8877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:25.380 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:25.379+0000 7faffe8877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:25.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:00:25.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:00:25.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:00:25.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:00:25.803 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:00:25.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:00:25.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:25.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:00:25.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:25.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:00:25.990 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:25.991+0000 7faffe8877c0 -1 Falling back to public interface 2024-04-25T08:00:26.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:27.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:27.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:27.312 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:00:27.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:00:27.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:27.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:00:27.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:28.478 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:00:28.479+0000 7faffe8877c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:00:28.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:28.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:28.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:00:28.797 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:00:28.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:28.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:00:29.413 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3606035060,v1:127.0.0.1:6811/3606035060] [v2:127.0.0.1:6812/3606035060,v1:127.0.0.1:6813/3606035060] exists,up eb1aa7b7-2c33-4d44-8889-234cd68aeca0 2024-04-25T08:00:29.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:00:29.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:00:29.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:00:29.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:00:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-04-25T08:00:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:00:29.995 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test1' already exists 2024-04-25T08:00:30.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:00:31.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T08:00:31.769 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T08:00:31.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:31.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-04-25T08:00:31.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T08:00:32.816 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test2' already exists 2024-04-25T08:00:32.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:00:33.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T08:00:34.576 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 2 size to 2 2024-04-25T08:00:34.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:34.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-04-25T08:00:34.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T08:00:35.156 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test3' already exists 2024-04-25T08:00:35.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:00:36.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-04-25T08:00:37.272 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 3 size to 2 2024-04-25T08:00:37.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-04-25T08:00:42.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:00:42.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:00:42.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:00:42.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:00:42.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:00:42.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:00:42.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:00:42.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:00:42.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:00:42.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:00:42.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:00:42.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:00:42.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:00:42.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:00:42.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:00:42.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:00:42.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:00:42.885 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T08:00:42.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:00:42.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:00:42.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:00:43.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-04-25T08:00:43.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-04-25T08:00:43.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-04-25T08:00:43.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:00:43.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:00:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672965 2024-04-25T08:00:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672965 2024-04-25T08:00:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-04-25T08:00:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:00:43.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-04-25T08:00:43.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:00:43.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:00:43.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-04-25T08:00:43.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:00:43.214 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836487 2024-04-25T08:00:43.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-04-25T08:00:43.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-04-25T08:00:43.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:00:43.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-04-25T08:00:43.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:00:44.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:00:44.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:00:45.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836487 2024-04-25T08:00:45.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:00:45.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672965 2024-04-25T08:00:45.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:00:45.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:00:45.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672965 2024-04-25T08:00:45.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:00:45.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672965 2024-04-25T08:00:45.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-04-25T08:00:45.197 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672965 2024-04-25T08:00:45.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:00:45.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672965 -lt 42949672965 2024-04-25T08:00:45.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:00:45.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:45.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:46.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:00:46.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:46.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:46.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:46.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:46.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:46.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:46.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:46.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:00:46.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:46.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:46.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:47.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:00:47.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:00:47.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:00:47.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:00:47.554 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:00:47.555 INFO:tasks.workunit.client.0.smithi031.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:00:37.226364+0000 0'0 27:20 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:00:34.999259+0000 0'0 2024-04-25T08:00:34.999259+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:54:20.361510+0000 0 0 2024-04-25T08:00:47.555 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:00:35.601204+0000 0'0 27:29 [0,1] 0 [0,1] 0 0'0 2024-04-25T08:00:32.346193+0000 0'0 2024-04-25T08:00:32.346193+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:51:22.698176+0000 0 0 2024-04-25T08:00:47.555 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:00:32.020113+0000 0'0 27:35 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:00:29.794273+0000 0'0 2024-04-25T08:00:29.794273+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:10:23.888589+0000 0 0 2024-04-25T08:00:47.555 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:00:47.555 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:00:47.555 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-04-25T08:00:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-04-25T08:00:47.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T08:00:47.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:00:47.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-04-25T08:00:47.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-04-25T08:00:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T08:00:48.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:00:48.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-04-25T08:00:48.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-04-25T08:00:48.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T08:00:48.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-04-25T08:00:48.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-04-25T08:00:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:00:48.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:00:49.137 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test3' does not exist 2024-04-25T08:00:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-04-25T08:00:49.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-04-25T08:00:49.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-04-25T08:00:49.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-04-25T08:00:49.157 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-04-25T08:00:49.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-04-25T08:00:49.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:00:49.957 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 1 2024-04-25T08:00:49.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-04-25T08:00:50.761 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 recovery_priority to 1 2024-04-25T08:00:50.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-04-25T08:00:51.581 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 2 size to 1 2024-04-25T08:00:51.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-04-25T08:00:52.377 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 2 recovery_priority to 2 2024-04-25T08:00:52.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:00:52.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:00:52.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:00:52.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:00:52.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:00:52.404 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:00:52.405 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:00:52.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:00:52.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:00:52.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:00:52.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:00:52.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:00:52.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:00:52.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:00:52.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:00:52.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:00:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:00:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T08:00:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:00:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:00:53.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:00:53.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836491 2024-04-25T08:00:53.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836491 2024-04-25T08:00:53.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491' 2024-04-25T08:00:53.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:00:53.234 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:00:53.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-04-25T08:00:53.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-04-25T08:00:53.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2024-04-25T08:00:53.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:00:53.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836491 2024-04-25T08:00:53.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:00:53.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:00:53.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836491 2024-04-25T08:00:53.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:00:53.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836491 2024-04-25T08:00:53.387 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836491 2024-04-25T08:00:53.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-04-25T08:00:53.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:00:53.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836489 -lt 21474836491 2024-04-25T08:00:53.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:00:54.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:00:54.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:00:55.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836491 2024-04-25T08:00:55.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:00:55.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-04-25T08:00:55.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:00:55.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:00:55.287 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-04-25T08:00:55.287 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:00:55.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-04-25T08:00:55.288 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672969 2024-04-25T08:00:55.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-25T08:00:55.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:00:55.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-25T08:00:55.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:00:55.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:55.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:56.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T08:00:56.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:56.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:56.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:56.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:56.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:56.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:56.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:56.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:00:56.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:56.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:56.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:57.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T08:00:57.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:00:57.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:00:57.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T08:00:57.358 INFO:tasks.workunit.client.0.smithi031.stderr:10+0 records in 2024-04-25T08:00:57.358 INFO:tasks.workunit.client.0.smithi031.stderr:10+0 records out 2024-04-25T08:00:57.359 INFO:tasks.workunit.client.0.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0406605 s, 258 MB/s 2024-04-25T08:00:57.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-04-25T08:00:57.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T08:00:57.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T08:00:57.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:57.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T08:00:57.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:57.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T08:00:58.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:58.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T08:00:58.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:58.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T08:00:58.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:58.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T08:00:59.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:59.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T08:00:59.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:00:59.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T08:01:00.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:00.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T08:01:00.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:00.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T08:01:00.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:00.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T08:01:01.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:01.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T08:01:01.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:01.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T08:01:01.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:01.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T08:01:02.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:02.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T08:01:02.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:02.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T08:01:02.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:02.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T08:01:03.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:03.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T08:01:03.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:03.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T08:01:04.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:04.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T08:01:04.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:04.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T08:01:04.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:04.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T08:01:05.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:05.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T08:01:05.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:05.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T08:01:05.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:05.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T08:01:06.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:06.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T08:01:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T08:01:06.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:06.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T08:01:07.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:07.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T08:01:07.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:07.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T08:01:08.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:08.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T08:01:08.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:08.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T08:01:08.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:08.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T08:01:09.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:09.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T08:01:09.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:09.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T08:01:09.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:09.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T08:01:10.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:10.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T08:01:10.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:10.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T08:01:11.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:11.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T08:01:11.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:11.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T08:01:11.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:11.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T08:01:12.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:12.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T08:01:12.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:12.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T08:01:12.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:12.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T08:01:13.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:13.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T08:01:13.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:13.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T08:01:13.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:13.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T08:01:14.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:14.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T08:01:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T08:01:15.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:15.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T08:01:15.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:15.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T08:01:15.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:15.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T08:01:16.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:16.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T08:01:16.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:16.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T08:01:17.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:17.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T08:01:17.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:17.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T08:01:17.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:17.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T08:01:18.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:18.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T08:01:18.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:18.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T08:01:19.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:19.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T08:01:19.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:19.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T08:01:19.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:19.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T08:01:20.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:20.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T08:01:20.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:20.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T08:01:20.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:20.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T08:01:21.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:21.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T08:01:21.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:21.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T08:01:22.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:22.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T08:01:22.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:22.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T08:01:22.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:22.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T08:01:23.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:23.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T08:01:23.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:23.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T08:01:23.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:23.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T08:01:24.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:24.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T08:01:24.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:24.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T08:01:25.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:25.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T08:01:25.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:25.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T08:01:25.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:25.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T08:01:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T08:01:26.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:26.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T08:01:27.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:27.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T08:01:27.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:27.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T08:01:27.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:27.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T08:01:28.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:28.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T08:01:28.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:28.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T08:01:29.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:29.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T08:01:29.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:29.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T08:01:29.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:29.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T08:01:30.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:30.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T08:01:30.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:30.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T08:01:31.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:31.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T08:01:31.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:31.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T08:01:31.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:31.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T08:01:32.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:32.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T08:01:32.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:32.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T08:01:33.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:33.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T08:01:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T08:01:33.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T08:01:34.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:34.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T08:01:34.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:34.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T08:01:34.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:34.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T08:01:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T08:01:35.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:35.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T08:01:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T08:01:36.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:36.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T08:01:36.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:36.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T08:01:37.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:37.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T08:01:37.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:37.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T08:01:37.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:37.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T08:01:38.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:38.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T08:01:38.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:38.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T08:01:38.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:38.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T08:01:39.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:39.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T08:01:39.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:39.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T08:01:39.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:39.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T08:01:40.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:40.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T08:01:40.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:40.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T08:01:41.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:41.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T08:01:41.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:41.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T08:01:41.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:41.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T08:01:42.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:42.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T08:01:42.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:42.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T08:01:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T08:01:43.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:43.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T08:01:43.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:43.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T08:01:44.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:44.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T08:01:44.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:44.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T08:01:44.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:44.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T08:01:45.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T08:01:45.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:45.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T08:01:45.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:45.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T08:01:46.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:46.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T08:01:46.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:46.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T08:01:47.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:47.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T08:01:47.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:47.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T08:01:47.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:47.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T08:01:48.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:48.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T08:01:48.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:48.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T08:01:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T08:01:49.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:49.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T08:01:49.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:49.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T08:01:50.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:50.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T08:01:50.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:50.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T08:01:51.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:51.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T08:01:51.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:51.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T08:01:51.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:51.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T08:01:52.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:52.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T08:01:52.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:52.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T08:01:52.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:52.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T08:01:53.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:53.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T08:01:53.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:53.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T08:01:54.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:54.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T08:01:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T08:01:54.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:54.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T08:01:55.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:55.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T08:01:55.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:55.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T08:01:56.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:56.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T08:01:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T08:01:56.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:56.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T08:01:57.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:57.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T08:01:57.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:57.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T08:01:57.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:57.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T08:01:58.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:58.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T08:01:58.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:58.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T08:01:59.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:59.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T08:01:59.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:59.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T08:01:59.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:01:59.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T08:02:00.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:00.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T08:02:00.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:00.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T08:02:00.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:00.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T08:02:01.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:01.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T08:02:01.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:01.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T08:02:02.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:02.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T08:02:02.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:02.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T08:02:02.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:02.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T08:02:03.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:03.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T08:02:03.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:03.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T08:02:04.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:04.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T08:02:04.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:04.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T08:02:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T08:02:05.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:05.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T08:02:05.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:05.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T08:02:06.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:06.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T08:02:06.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:06.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T08:02:06.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:06.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T08:02:07.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:07.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T08:02:07.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:07.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T08:02:07.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:07.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T08:02:08.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:08.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T08:02:08.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:08.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T08:02:09.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:09.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T08:02:09.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:09.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T08:02:09.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:09.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T08:02:10.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:10.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T08:02:10.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:10.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T08:02:11.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:11.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T08:02:11.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:11.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T08:02:11.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:11.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T08:02:12.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:12.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T08:02:12.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T08:02:12.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-04-25T08:02:12.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T08:02:12.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T08:02:12.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:12.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T08:02:12.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:12.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T08:02:13.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:13.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T08:02:13.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:13.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T08:02:13.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:13.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T08:02:14.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:14.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T08:02:14.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:14.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T08:02:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T08:02:15.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:15.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T08:02:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T08:02:16.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:16.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T08:02:16.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:16.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T08:02:16.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:16.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T08:02:17.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:17.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T08:02:17.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:17.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T08:02:18.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:18.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T08:02:18.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:18.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T08:02:18.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:18.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T08:02:19.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:19.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T08:02:19.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:19.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T08:02:19.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:19.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T08:02:20.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:20.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T08:02:20.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:20.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T08:02:21.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:21.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T08:02:21.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:21.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T08:02:21.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:21.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T08:02:22.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:22.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T08:02:22.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:22.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T08:02:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T08:02:23.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:23.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T08:02:23.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:23.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T08:02:23.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:23.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T08:02:24.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:24.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T08:02:24.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:24.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T08:02:25.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:25.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T08:02:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T08:02:25.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:25.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T08:02:26.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:26.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T08:02:26.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:26.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T08:02:26.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:26.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T08:02:27.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:27.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T08:02:27.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:27.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T08:02:28.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:28.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T08:02:28.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:28.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T08:02:28.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:28.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T08:02:29.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:29.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T08:02:29.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:29.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T08:02:29.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:29.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T08:02:30.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:30.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T08:02:30.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:30.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T08:02:31.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:31.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T08:02:31.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:31.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T08:02:31.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:31.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T08:02:32.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:32.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T08:02:32.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:32.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T08:02:33.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:33.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T08:02:33.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:33.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T08:02:33.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:33.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T08:02:34.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:34.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T08:02:34.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:34.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T08:02:34.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:34.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T08:02:35.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:35.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T08:02:35.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:35.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T08:02:36.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:36.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T08:02:36.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:36.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T08:02:36.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:36.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T08:02:37.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:37.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T08:02:37.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:37.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T08:02:37.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:37.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T08:02:38.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:38.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T08:02:38.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:38.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T08:02:39.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:39.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T08:02:39.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:39.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T08:02:39.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:39.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T08:02:40.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:40.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T08:02:40.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:40.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T08:02:40.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:40.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T08:02:41.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:41.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T08:02:41.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:41.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T08:02:41.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:41.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T08:02:42.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:42.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T08:02:42.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:42.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T08:02:43.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:43.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T08:02:43.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:43.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T08:02:43.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:43.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T08:02:44.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:44.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T08:02:44.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:44.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T08:02:44.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:44.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T08:02:45.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:45.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T08:02:45.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:45.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T08:02:45.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:45.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T08:02:46.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:46.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T08:02:46.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:46.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T08:02:47.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:47.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T08:02:47.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:47.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T08:02:48.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:48.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T08:02:48.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:48.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T08:02:49.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:49.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T08:02:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T08:02:49.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:49.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T08:02:50.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:50.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T08:02:50.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:50.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T08:02:51.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:51.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T08:02:51.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:51.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T08:02:51.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:51.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T08:02:52.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:52.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T08:02:52.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:52.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T08:02:52.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:52.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T08:02:53.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:53.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T08:02:53.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:53.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T08:02:54.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:54.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T08:02:54.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:54.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T08:02:54.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:54.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T08:02:55.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:55.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T08:02:55.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:55.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T08:02:56.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:56.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T08:02:56.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:56.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T08:02:56.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:56.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T08:02:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:57.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T08:02:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T08:02:57.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:57.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T08:02:58.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:58.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T08:02:58.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:58.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T08:02:59.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:59.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T08:02:59.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:59.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T08:02:59.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:02:59.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T08:03:00.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:00.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T08:03:00.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:00.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T08:03:00.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:00.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T08:03:01.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:01.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T08:03:01.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:01.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T08:03:01.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:01.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T08:03:02.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:02.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T08:03:02.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:02.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T08:03:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T08:03:03.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:03.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T08:03:03.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:03.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T08:03:04.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:04.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T08:03:04.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:04.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T08:03:04.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:04.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T08:03:05.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:05.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T08:03:05.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:05.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T08:03:06.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:06.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T08:03:06.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:06.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T08:03:06.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:06.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T08:03:07.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:07.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T08:03:07.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:07.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T08:03:07.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:07.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T08:03:08.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:08.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T08:03:08.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:08.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T08:03:09.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:09.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T08:03:09.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:09.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T08:03:09.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:09.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T08:03:10.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:10.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T08:03:10.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:10.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T08:03:11.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:11.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T08:03:11.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:11.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T08:03:11.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:11.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T08:03:12.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:12.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T08:03:12.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:12.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T08:03:13.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:13.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T08:03:13.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:13.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T08:03:13.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:13.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T08:03:14.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:14.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T08:03:14.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:14.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T08:03:14.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:14.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T08:03:15.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:15.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T08:03:15.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:15.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T08:03:15.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:15.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T08:03:16.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:16.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T08:03:16.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:16.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T08:03:17.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:17.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T08:03:17.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:17.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T08:03:17.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:17.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T08:03:18.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:18.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T08:03:18.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:18.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T08:03:18.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:18.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T08:03:19.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:19.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T08:03:19.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:19.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T08:03:19.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:19.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T08:03:20.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:20.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T08:03:20.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:20.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T08:03:21.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:21.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T08:03:21.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:21.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T08:03:22.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:22.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T08:03:22.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:22.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T08:03:23.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:23.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T08:03:23.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:23.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T08:03:23.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:23.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T08:03:24.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:24.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T08:03:24.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T08:03:25.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:25.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T08:03:25.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:25.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T08:03:25.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:25.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T08:03:26.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:26.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T08:03:26.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:26.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T08:03:27.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:27.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T08:03:27.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:27.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T08:03:27.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:03:27.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T08:03:28.267 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-04-25T08:03:28.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-04-25T08:03:28.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-04-25T08:03:28.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T08:03:28.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T08:03:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T08:03:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T08:03:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T08:03:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:03:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:03:28.733 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:03:28.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:03:28.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:03:29.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-04-25T08:03:29.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:03:29.639 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:03:29.639 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 190 0 0 0 0 1986002944 0 0 568 0 568 active+clean 2024-04-25T08:00:51.777451+0000 36'568 36:618 [0] 0 [0] 0 0'0 2024-04-25T08:00:32.346193+0000 0'0 2024-04-25T08:00:32.346193+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:38:54.617442+0000 0 0 2024-04-25T08:03:29.639 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:00:50.153224+0000 36'600 36:658 [1] 1 [1] 1 0'0 2024-04-25T08:00:29.794273+0000 0'0 2024-04-25T08:00:29.794273+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:41:52.788345+0000 0 0 2024-04-25T08:03:29.639 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:03:29.639 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:03:29.639 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:03:29.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-04-25T08:03:29.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T08:03:30.362 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T08:03:30.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T08:03:31.187 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 2 size to 2 2024-04-25T08:03:31.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:03:31.727 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:03:31.727 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:00:51.777451+0000 36'600 36:650 [0] 0 [0] 0 0'0 2024-04-25T08:00:32.346193+0000 0'0 2024-04-25T08:00:32.346193+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:38:54.617442+0000 0 0 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:00:50.153224+0000 36'600 36:658 [1] 1 [1] 1 0'0 2024-04-25T08:00:29.794273+0000 0'0 2024-04-25T08:00:29.794273+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:41:52.788345+0000 0 0 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:03:31.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-04-25T08:03:31.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-04-25T08:03:31.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:03:31.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:03:31.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:03:31.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:03:32.155 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:03:32.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-04-25T08:03:32.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T08:03:34.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-04-25T08:03:34.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-04-25T08:03:34.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-04-25T08:03:34.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:03:34.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:03:34.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:03:34.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:03:34.599 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:03:34.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 1 == 2 2024-04-25T08:03:34.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T08:03:36.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2024-04-25T08:03:36.620 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T08:03:36.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2024-04-25T08:03:36.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:03:36.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:03:36.625 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:03:36.625 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:03:37.036 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:03:37.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-04-25T08:03:37.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-04-25T08:03:37.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-04-25T08:03:37.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:03:37.464 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:03:37.465 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T08:03:33.212448+0000 36'600 43:667 [0,1] 0 [0] 0 0'0 2024-04-25T08:00:32.346193+0000 0'0 2024-04-25T08:00:32.346193+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:03:37.465 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 195 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T08:03:31.519924+0000 36'600 43:683 [1,0] 1 [1] 1 0'0 2024-04-25T08:00:29.794273+0000 0'0 2024-04-25T08:00:29.794273+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:03:37.465 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:03:37.465 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:03:37.465 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:03:37.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-04-25T08:03:37.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:03:37.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:03:37.484 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:03:37.484 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:37.484 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:03:37.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:03:37.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T08:03:37.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:37.610 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 2024-04-25T08:03:37.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-04-25T08:03:37.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:03:37.611 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:03:37.611 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:03:37.611 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:03:37.611 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:03:37.611 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 191, 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:03:37.612 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "2.0", 2024-04-25T08:03:37.613 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 192, 2024-04-25T08:03:37.613 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:03:37.613 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:03:37.613 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:03:37.613 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:03:37.613 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:03:37.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-04-25T08:03:37.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:03:37.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:03:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:03:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:03:37.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.0.asok 2024-04-25T08:03:37.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T08:03:37.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.0.asok dump_recovery_reservations 2024-04-25T08:03:37.738 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 2024-04-25T08:03:37.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-04-25T08:03:37.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:03:37.739 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:03:37.739 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:03:37.739 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:03:37.739 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "2.0", 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 192, 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:03:37.740 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 191, 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:03:37.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:03:37.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T08:03:37.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T08:03:37.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T08:03:37.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:03:37.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:03:37.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-04-25T08:03:37.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T08:03:37.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T08:03:37.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:03:37.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T08:03:37.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T08:03:37.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T08:03:37.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:03:37.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T08:03:37.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-04-25T08:03:37.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T08:03:37.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:03:37.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:03:37.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T08:03:37.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T08:03:37.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T08:03:37.925 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:03:37.925 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:03:37.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-04-25T08:03:37.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T08:03:37.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:03:37.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T08:03:37.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T08:03:37.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T08:03:37.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T08:03:37.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:03:37.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T08:03:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-04-25T08:03:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T08:03:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:03:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:03:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:03:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:03:38.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:03:38.032 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:03:38.033 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:03:38.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:03:38.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:03:38.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:03:38.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:03:38.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:03:38.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:03:38.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:03:38.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:03:38.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:03:38.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:03:38.686 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T08:03:38.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:03:38.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:03:38.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:03:38.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-04-25T08:03:38.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-04-25T08:03:38.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-04-25T08:03:38.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:03:38.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:03:39.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673004 2024-04-25T08:03:39.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673004 2024-04-25T08:03:39.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673004' 2024-04-25T08:03:39.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:03:39.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-04-25T08:03:39.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:03:39.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:03:39.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-04-25T08:03:39.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:03:39.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-04-25T08:03:39.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-04-25T08:03:39.194 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836526 2024-04-25T08:03:39.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:03:39.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836524 -lt 21474836526 2024-04-25T08:03:39.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:03:40.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:03:40.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:03:41.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836526 2024-04-25T08:03:41.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:03:41.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673004 2024-04-25T08:03:41.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:03:41.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:03:41.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673004 2024-04-25T08:03:41.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:03:41.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673004 2024-04-25T08:03:41.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-04-25T08:03:41.390 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673004 2024-04-25T08:03:41.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:03:41.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673005 -lt 42949673004 2024-04-25T08:03:41.965 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:03:41.965 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:41.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:42.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T08:03:42.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:42.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:42.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:42.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:42.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:42.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:42.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:43.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:43.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:43.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:43.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:03:43.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:03:43.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:43.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:44.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:44.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:44.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:44.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:44.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:44.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:44.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:44.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:44.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:44.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:44.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:44.644 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:45.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:45.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:45.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24815728 2024-04-25T08:03:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24815728 '!=' null 2024-04-25T08:03:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:45.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:45.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:45.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:45.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:45.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:45.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:45.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:45.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:46.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:46.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:46.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:46.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:46.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:46.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:46.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701290 2024-04-25T08:03:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701290 '!=' null 2024-04-25T08:03:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:48.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:48.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:48.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:48.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:48.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:48.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:48.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:48.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:48.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:48.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:48.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:48.698 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:49.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:49.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:49.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:50.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25334110 2024-04-25T08:03:50.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25334110 '!=' null 2024-04-25T08:03:50.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:50.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:50.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:50.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:50.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:50.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:50.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:50.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:50.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:50.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:50.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:50.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:50.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:50.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:50.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:51.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:51.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:51.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:51.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36691123 2024-04-25T08:03:51.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36691123 '!=' null 2024-04-25T08:03:51.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:51.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:51.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:51.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:51.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:51.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:51.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:51.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:51.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:51.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:51.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:52.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:52.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:52.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:52.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:52.829 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:53.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461003 2024-04-25T08:03:53.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461003 '!=' null 2024-04-25T08:03:53.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:53.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:53.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:53.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:53.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:53.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:53.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:53.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:53.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:53.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:53.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:54.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:54.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:54.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:54.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:54.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:54.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:54.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:55.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30576314 2024-04-25T08:03:55.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30576314 '!=' null 2024-04-25T08:03:55.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:55.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:55.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:55.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:55.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:55.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:55.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:56.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:56.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:56.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:56.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:56.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:56.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:56.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:56.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:56.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:56.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:56.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:56.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:56.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:57.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701733 2024-04-25T08:03:57.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701733 '!=' null 2024-04-25T08:03:57.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:57.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:57.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:57.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:57.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:57.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:57.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:57.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:57.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:57.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:57.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:57.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:57.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:57.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:57.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:58.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:58.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:59.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586221 2024-04-25T08:03:59.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586221 '!=' null 2024-04-25T08:03:59.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:59.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:03:59.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:59.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:59.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:59.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:59.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:59.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:59.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:59.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:59.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:03:59.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:59.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:59.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:00.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:00.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:01.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34941902 2024-04-25T08:04:01.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34941902 '!=' null 2024-04-25T08:04:01.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:01.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:01.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:01.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:01.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:01.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:01.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:01.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:01.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:02.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:02.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:02.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:02.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:02.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:02.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:02.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:02.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:02.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585951 2024-04-25T08:04:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585951 '!=' null 2024-04-25T08:04:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:03.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:03.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:03.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:03.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:03.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:03.523 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:03.523 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:03.523 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:03.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:03.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:03.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:03.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:04.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:04.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:04.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:05.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29700636 2024-04-25T08:04:05.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29700636 '!=' null 2024-04-25T08:04:05.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:05.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:05.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:05.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:05.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:05.667 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:05.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:05.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:05.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:05.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:05.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:06.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:06.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:06.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:06.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:06.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:06.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:06.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:07.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701060 2024-04-25T08:04:07.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701060 '!=' null 2024-04-25T08:04:07.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:07.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:07.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:07.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:07.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:07.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:07.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:07.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:07.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:07.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:07.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:08.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:08.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:08.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:08.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:08.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:08.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:08.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:08.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:08.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:08.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:08.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:08.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:08.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:09.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586472 2024-04-25T08:04:09.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586472 '!=' null 2024-04-25T08:04:09.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:09.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:09.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:09.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:09.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:09.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:09.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:09.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:09.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:09.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:09.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:09.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:09.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:09.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:09.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:10.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:11.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35817099 2024-04-25T08:04:11.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35817099 '!=' null 2024-04-25T08:04:11.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:11.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:11.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:11.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:11.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:11.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:11.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:11.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:11.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:11.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:11.199 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:11.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:11.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:11.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:11.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:12.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:12.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:12.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:12.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:12.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:12.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:12.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:12.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:12.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:12.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460386 2024-04-25T08:04:12.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460386 '!=' null 2024-04-25T08:04:12.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:12.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:12.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:13.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:13.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:13.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:13.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:13.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:13.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:13.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:13.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:13.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:13.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:13.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:13.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:14.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460386 2024-04-25T08:04:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460386 '!=' null 2024-04-25T08:04:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:14.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:14.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:14.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:14.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:14.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:14.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:14.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:14.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:14.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:15.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:15.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:15.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:15.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:15.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:16.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575098 2024-04-25T08:04:16.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575098 '!=' null 2024-04-25T08:04:16.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:16.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:16.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:16.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:16.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:16.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:16.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:16.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:16.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:16.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:16.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:17.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:17.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:17.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:17.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:17.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:17.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:17.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:17.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:17.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:17.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:17.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:17.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:17.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:18.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574830 2024-04-25T08:04:18.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574830 '!=' null 2024-04-25T08:04:18.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:18.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:18.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:18.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:18.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:18.622 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:18.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:18.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:18.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:18.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:18.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:19.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:19.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:19.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:19.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:19.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:19.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:19.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:20.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459870 2024-04-25T08:04:20.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459870 '!=' null 2024-04-25T08:04:20.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:20.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:20.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:20.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:20.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:20.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:20.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:20.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:20.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:20.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:20.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:20.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:20.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:20.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:20.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:21.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:21.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:22.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36689895 2024-04-25T08:04:22.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36689895 '!=' null 2024-04-25T08:04:22.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:22.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:22.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:22.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:22.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:22.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:22.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:22.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:22.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:22.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:22.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:22.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:22.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:22.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:22.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:23.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:23.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459692 2024-04-25T08:04:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459692 '!=' null 2024-04-25T08:04:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:24.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:24.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:24.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:24.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:24.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:24.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:24.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:24.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:24.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:24.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:24.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:24.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:25.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:25.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:25.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574706 2024-04-25T08:04:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574706 '!=' null 2024-04-25T08:04:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:26.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:26.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:26.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:26.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:26.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:26.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:26.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:26.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:26.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:26.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:26.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:26.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:26.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:27.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:27.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:27.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:27.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574780 2024-04-25T08:04:27.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574780 '!=' null 2024-04-25T08:04:27.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:27.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:27.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:28.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:28.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:28.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:28.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:28.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:28.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:28.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:28.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:28.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:28.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:28.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:28.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:29.239 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:29.239 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460069 2024-04-25T08:04:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460069 '!=' null 2024-04-25T08:04:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:30.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:30.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:30.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:30.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:30.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:30.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:30.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:30.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:30.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:30.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:30.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:30.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:31.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:31.476 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:31.476 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36689197 2024-04-25T08:04:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36689197 '!=' null 2024-04-25T08:04:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:32.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:32.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:32.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:32.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:32.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:32.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:32.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:32.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:32.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:32.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:32.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:33.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:33.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:33.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:33.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:33.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:33.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:33.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:33.523 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:33.523 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459491 2024-04-25T08:04:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459491 '!=' null 2024-04-25T08:04:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:34.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:34.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:34.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:34.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:34.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:34.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:34.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:34.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:34.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:34.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:34.721 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:34.722 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:34.722 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:35.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:35.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:35.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:35.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:35.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:35.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:35.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:35.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:35.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:36.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701394 2024-04-25T08:04:36.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701394 '!=' null 2024-04-25T08:04:36.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:36.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:36.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:36.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:36.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:36.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:36.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:36.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:36.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:36.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:36.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:36.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:36.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:36.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:36.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:37.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:37.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701134 2024-04-25T08:04:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701134 '!=' null 2024-04-25T08:04:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:37.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:37.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:37.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:37.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:37.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:37.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:37.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:37.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:38.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:38.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:38.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:38.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:39.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:39.145 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:39.145 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:39.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586090 2024-04-25T08:04:39.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586090 '!=' null 2024-04-25T08:04:39.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:39.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:39.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:39.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:39.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:39.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:39.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:39.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:39.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:39.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:39.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:40.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:40.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:40.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:40.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:40.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:40.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:40.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:40.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:40.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:40.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:40.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:40.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:40.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:41.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34942205 2024-04-25T08:04:41.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34942205 '!=' null 2024-04-25T08:04:41.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:41.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:41.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:41.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:41.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:41.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:41.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:41.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:41.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:41.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:41.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:42.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:42.206 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:42.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:42.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:42.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:42.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:43.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713005 2024-04-25T08:04:43.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713005 '!=' null 2024-04-25T08:04:43.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:43.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:43.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:43.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:43.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:43.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:43.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:43.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:43.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:43.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:43.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:44.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:44.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:44.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:44.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:44.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:44.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:45.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827896 2024-04-25T08:04:45.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827896 '!=' null 2024-04-25T08:04:45.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:45.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:45.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:45.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:45.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:45.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:45.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:45.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:45.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:45.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:45.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:45.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:45.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:45.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:45.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:46.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:46.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:46.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:46.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:46.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:46.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:46.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:46.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:46.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:47.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574760 2024-04-25T08:04:47.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574760 '!=' null 2024-04-25T08:04:47.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:47.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:47.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:47.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:47.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:47.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:47.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:47.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:47.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:47.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:47.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:47.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:47.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:47.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:47.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:48.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:48.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:48.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460349 2024-04-25T08:04:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460349 '!=' null 2024-04-25T08:04:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:49.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:49.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:49.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:49.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:49.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:49.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:49.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:49.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:49.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:50.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:50.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:51.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=37564943 2024-04-25T08:04:51.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 37564943 '!=' null 2024-04-25T08:04:51.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:51.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:51.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:51.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:51.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:51.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:51.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:51.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:51.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:51.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:51.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:51.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:51.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:51.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:51.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:52.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:52.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:52.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:52.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:52.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:52.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:52.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:52.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:52.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:52.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26208374 2024-04-25T08:04:52.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26208374 '!=' null 2024-04-25T08:04:52.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:52.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:52.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:53.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:53.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:53.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:53.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:53.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:53.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:53.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:53.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:53.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:53.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:53.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:53.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:54.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:54.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:54.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:54.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26208374 2024-04-25T08:04:54.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26208374 '!=' null 2024-04-25T08:04:54.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:54.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:54.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:54.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:54.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:54.973 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:54.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:54.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:54.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:54.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:54.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:55.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:55.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:55.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:56.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:56.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:56.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:56.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31449193 2024-04-25T08:04:56.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31449193 '!=' null 2024-04-25T08:04:56.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:56.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:56.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:56.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:56.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:56.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:56.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:56.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:56.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:56.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:56.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:57.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:57.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:57.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:57.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:58.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:58.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31449193 2024-04-25T08:04:58.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31449193 '!=' null 2024-04-25T08:04:58.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:58.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:04:58.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:58.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:58.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:58.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:58.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:58.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:58.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:58.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:58.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:59.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:04:59.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:59.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:59.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:59.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:59.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:00.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460363 2024-04-25T08:05:00.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460363 '!=' null 2024-04-25T08:05:00.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:00.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:00.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:00.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:00.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:00.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:00.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:00.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:00.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:00.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:00.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:01.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:01.043 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:01.043 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:01.043 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:01.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:01.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:01.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:01.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:01.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:01.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:01.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:01.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:01.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:02.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36689957 2024-04-25T08:05:02.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36689957 '!=' null 2024-04-25T08:05:02.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:02.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:02.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:02.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:02.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:02.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:02.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:02.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:02.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:02.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:02.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:02.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:03.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:03.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:04.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586267 2024-04-25T08:05:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586267 '!=' null 2024-04-25T08:05:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:04.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:04.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:04.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:04.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:04.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:04.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:04.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:04.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:05.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:05.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:05.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:05.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:05.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:05.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:05.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:05.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:05.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:05.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701217 2024-04-25T08:05:05.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701217 '!=' null 2024-04-25T08:05:05.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:05.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:05.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:06.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:06.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:06.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:06.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:06.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:06.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:06.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:06.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:06.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:06.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:06.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:06.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:07.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:07.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:07.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575200 2024-04-25T08:05:07.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575200 '!=' null 2024-04-25T08:05:07.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:07.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:07.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:07.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:07.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:07.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:07.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:07.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:07.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:07.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:07.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:08.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:08.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:08.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:08.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:09.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:09.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:10.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460387 2024-04-25T08:05:10.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460387 '!=' null 2024-04-25T08:05:10.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:10.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:10.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:10.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:10.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:10.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:10.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:10.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:10.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:10.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:10.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:10.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:10.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:10.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:10.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:11.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:11.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36690517 2024-04-25T08:05:11.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36690517 '!=' null 2024-04-25T08:05:11.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:11.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:11.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:12.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:12.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:12.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:12.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:12.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:12.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:12.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:12.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:12.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:12.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:12.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:12.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:13.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:13.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:13.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460091 2024-04-25T08:05:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460091 '!=' null 2024-04-25T08:05:13.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:13.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:13.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:13.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:13.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:13.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:13.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:13.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:13.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:13.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:13.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:14.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:14.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:14.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:14.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:15.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:15.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:15.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:15.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:15.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:15.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:15.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:15.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:15.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:15.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575074 2024-04-25T08:05:15.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575074 '!=' null 2024-04-25T08:05:15.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:15.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:15.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:15.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:15.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:15.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:15.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:15.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:15.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:15.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:15.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:16.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:16.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:16.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:16.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:16.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:16.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:17.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701572 2024-04-25T08:05:17.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701572 '!=' null 2024-04-25T08:05:17.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:17.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:17.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:17.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:17.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:17.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:17.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:17.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:17.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:17.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:17.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:18.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:18.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:18.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:18.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:18.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:18.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:18.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:19.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586685 2024-04-25T08:05:19.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586685 '!=' null 2024-04-25T08:05:19.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:19.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:19.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:19.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:19.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:19.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:19.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:19.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:19.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:19.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:19.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:20.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:20.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:20.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:20.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:20.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:20.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:20.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:21.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35816502 2024-04-25T08:05:21.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35816502 '!=' null 2024-04-25T08:05:21.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:21.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:21.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:21.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:21.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:21.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:21.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:21.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:21.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:21.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:21.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:22.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:22.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:22.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:22.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:22.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:22.840 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:22.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:23.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586496 2024-04-25T08:05:23.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586496 '!=' null 2024-04-25T08:05:23.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:23.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:23.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:23.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:23.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:23.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:23.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:23.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:23.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:23.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:23.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:24.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:24.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:24.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:24.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:24.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:24.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:25.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701447 2024-04-25T08:05:25.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701447 '!=' null 2024-04-25T08:05:25.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:25.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:25.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:25.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:25.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:25.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:25.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:25.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:25.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:25.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:25.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:26.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:26.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:26.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:26.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:26.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:26.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:26.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701087 2024-04-25T08:05:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701087 '!=' null 2024-04-25T08:05:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:27.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:27.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:27.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:27.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:27.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:27.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:27.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:27.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:27.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:28.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:28.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:28.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:28.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:28.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:28.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:29.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459303 2024-04-25T08:05:29.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459303 '!=' null 2024-04-25T08:05:29.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:29.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:29.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:29.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:29.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:29.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:29.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:29.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:29.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:29.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:29.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:30.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:30.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:30.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:30.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:30.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:30.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:31.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34941634 2024-04-25T08:05:31.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34941634 '!=' null 2024-04-25T08:05:31.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:31.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:31.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:31.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:31.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:31.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:31.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:31.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:31.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:31.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:31.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:32.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:32.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:32.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:32.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:32.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:32.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:32.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:32.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:32.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:32.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:32.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:32.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:32.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:33.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712049 2024-04-25T08:05:33.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712049 '!=' null 2024-04-25T08:05:33.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:33.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:33.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:33.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:33.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:33.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:33.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:33.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:33.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:33.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:33.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:33.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:33.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:33.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:33.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:34.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:34.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:34.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:35.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827003 2024-04-25T08:05:35.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827003 '!=' null 2024-04-25T08:05:35.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:35.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:35.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:35.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:35.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:35.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:35.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:35.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:35.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:35.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:35.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:35.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:35.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:35.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:35.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:36.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:36.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:36.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:36.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827853 2024-04-25T08:05:36.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827853 '!=' null 2024-04-25T08:05:36.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:36.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:36.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:37.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:37.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:37.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:37.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:37.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:37.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:37.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:37.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:37.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:37.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:37.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:37.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:38.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:38.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:38.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:38.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:38.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:38.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:38.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:38.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:38.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:38.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827853 2024-04-25T08:05:38.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827853 '!=' null 2024-04-25T08:05:38.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:38.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:38.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:38.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:38.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:38.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:38.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:38.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:38.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:38.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:38.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:39.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:39.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:39.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:39.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:40.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:40.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:40.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712964 2024-04-25T08:05:40.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712964 '!=' null 2024-04-25T08:05:40.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:40.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:40.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:40.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:40.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:40.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:40.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:40.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:40.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:40.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:40.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:41.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:41.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:41.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:41.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:41.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:41.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:41.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:42.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34069041 2024-04-25T08:05:42.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34069041 '!=' null 2024-04-25T08:05:42.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:42.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:42.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:42.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:42.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:42.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:42.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:42.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:42.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:42.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:42.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:43.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:43.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:43.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:43.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:43.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:43.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:43.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585980 2024-04-25T08:05:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585980 '!=' null 2024-04-25T08:05:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:44.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:44.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:44.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:44.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:44.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:44.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:44.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:44.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:45.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:45.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:45.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:45.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:45.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:45.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:45.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:45.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:46.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701339 2024-04-25T08:05:46.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701339 '!=' null 2024-04-25T08:05:46.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:46.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:46.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:46.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:46.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:46.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:46.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:46.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:46.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:46.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:46.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:46.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:46.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:46.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:46.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:47.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:47.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:47.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:47.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:47.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:47.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:47.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:47.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:47.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:48.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827315 2024-04-25T08:05:48.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827315 '!=' null 2024-04-25T08:05:48.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:48.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:48.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:48.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:48.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:48.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:48.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:48.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:48.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:48.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:48.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:48.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:48.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:48.791 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:48.791 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:49.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:49.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:49.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:50.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712592 2024-04-25T08:05:50.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712592 '!=' null 2024-04-25T08:05:50.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:50.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:50.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:50.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:50.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:50.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:50.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:50.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:50.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:50.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:50.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:50.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:50.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:50.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:50.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:51.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:51.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:51.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:52.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34068750 2024-04-25T08:05:52.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34068750 '!=' null 2024-04-25T08:05:52.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:52.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:52.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:52.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:52.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:52.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:52.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:52.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:52.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:52.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:52.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:52.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:52.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:52.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:52.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:53.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:53.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:53.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:53.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712407 2024-04-25T08:05:53.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712407 '!=' null 2024-04-25T08:05:53.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:53.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:53.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:53.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:53.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:53.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:53.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:53.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:53.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:53.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:53.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:54.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:54.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:54.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:54.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:55.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:55.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:55.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:55.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:55.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:55.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:55.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:55.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:55.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:56.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827549 2024-04-25T08:05:56.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827549 '!=' null 2024-04-25T08:05:56.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:56.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:56.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:56.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:56.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:56.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:56.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:56.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:56.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:56.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:56.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:56.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:56.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:56.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:56.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:57.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:57.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:05:57.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:58.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953588 2024-04-25T08:05:58.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953588 '!=' null 2024-04-25T08:05:58.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:05:58.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:05:58.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:05:58.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:05:58.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:05:58.217 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:05:58.217 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:05:58.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:05:58.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:05:58.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:05:58.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:05:58.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:05:58.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:05:58.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:05:58.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:05:59.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:05:59.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:05:59.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:06:00.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712774 2024-04-25T08:06:00.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712774 '!=' null 2024-04-25T08:06:00.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:06:00.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:06:00.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:06:00.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:06:00.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:06:00.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:06:00.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:06:00.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:06:00.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:06:00.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:06:00.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:06:00.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:06:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:06:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:06:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:06:01.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:06:01.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:06:01.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:06:01.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:06:01.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:06:01.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:06:01.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:06:01.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:06:01.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:06:01.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:06:01.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:06:01.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:06:01.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:06:01.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:06:01.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:06:01.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:06:01.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:06:01.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:06:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:06:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:06:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:06:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:06:02.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:06:02.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:06:02.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:06:02.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:06:02.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:06:03.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207353 2024-04-25T08:06:03.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207353 '!=' null 2024-04-25T08:06:03.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:06:03.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:06:03.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:06:03.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:06:03.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:06:03.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:06:03.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:06:03.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:06:03.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:06:03.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:06:03.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:06:03.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:06:03.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:06:03.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:06:03.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:06:04.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:06:04.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:06:04.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:06:04.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:06:04.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:06:04.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:06:04.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:06:04.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:06:04.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:06:04.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207353 2024-04-25T08:06:04.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207353 '!=' null 2024-04-25T08:06:04.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:06:04.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:06:04.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:06:04.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:06:04.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:06:04.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:06:04.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:06:04.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:06:04.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:06:04.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:06:04.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:06:05.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:06:05.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:06:05.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:06:05.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:06:05.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T08:06:05.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:06:05.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:06:05.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:06:05.770 INFO:tasks.workunit.client.0.smithi031.stdout:TEST PASSED 2024-04-25T08:06:05.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-04-25T08:06:05.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-04-25T08:06:05.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:06:05.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:06:06.341 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test1' does not exist 2024-04-25T08:06:06.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-04-25T08:06:06.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T08:06:06.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T08:06:07.008 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test2' does not exist 2024-04-25T08:06:07.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:06:07.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:06:07.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:06:07.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:06:07.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:06:07.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:06:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:06:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-04-25T08:06:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:06:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:06:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:06:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:06:12.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:06:12.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:06:12.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:06:12.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:06:12.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:06:12.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:06:12.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:06:12.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:06:12.356 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:06:12.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:06:12.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:06:12.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:06:12.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:12.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:06:12.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:06:12.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:12.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:06:12.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:06:12.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:06:12.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:06:12.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:12.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:12.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202871 2024-04-25T08:06:12.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:06:12.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:06:12.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T08:06:12.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T08:06:12.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:06:12.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:06:12.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:06:12.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:06:12.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:06:12.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:06:12.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:06:12.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:06:12.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:12.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:06:12.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:06:12.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:12.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:06:12.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:06:12.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:06:12.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:06:12.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:12.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:12.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202871 2024-04-25T08:06:12.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:06:12.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:06:12.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T08:06:12.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:06:12.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:12.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:12.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.202871 2024-04-25T08:06:12.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:06:12.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:06:12.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:06:12.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T08:06:12.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-04-25T08:06:12.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T08:06:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T08:06:12.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:06:13.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:06:13.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:06:13.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:06:13.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:06:13.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:06:13.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:06:13.129 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:06:13.129 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:06:13.129 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:06:13.130 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:06:13.130 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:13.130 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:13.130 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-mon.a.asok 2024-04-25T08:06:13.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:06:13.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202871/ceph-mon.a.asok config get fsid 2024-04-25T08:06:13.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:06:13.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:06:13.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:06:13.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:06:13.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:06:13.251 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:06:13.251 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:06:13.251 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:06:13.252 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:06:13.252 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:13.252 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:13.252 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-mon.a.asok 2024-04-25T08:06:13.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:06:13.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202871/ceph-mon.a.asok config get mon_host 2024-04-25T08:06:13.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-04-25T08:06:13.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T08:06:13.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:06:13.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:06:13.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:06:13.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T08:06:13.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:06:14.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:06:14.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:14.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:14.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:14.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:14.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:14.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:14.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:06:14.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:06:14.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-04-25T08:06:14.316 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-04-25T08:06:14.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T08:06:14.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:06:14.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:06:14.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2fc72a75-fcb8-4d2b-976b-cdb000f47c67 2024-04-25T08:06:14.324 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 2fc72a75-fcb8-4d2b-976b-cdb000f47c67 2024-04-25T08:06:14.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2fc72a75-fcb8-4d2b-976b-cdb000f47c67' 2024-04-25T08:06:14.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:06:14.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2Dipmzpl8FBAAk5DfXcACTZFxBeUoYp9WQw== 2024-04-25T08:06:14.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2Dipmzpl8FBAAk5DfXcACTZFxBeUoYp9WQw=="}' 2024-04-25T08:06:14.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2fc72a75-fcb8-4d2b-976b-cdb000f47c67 -i td/osd-recovery-prio/0/new.json 2024-04-25T08:06:14.665 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:06:14.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T08:06:14.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2Dipmzpl8FBAAk5DfXcACTZFxBeUoYp9WQw== --osd-uuid 2fc72a75-fcb8-4d2b-976b-cdb000f47c67 2024-04-25T08:06:14.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.718+0000 7f643dcd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:14.742 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.742+0000 7f643dcd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:14.744 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.742+0000 7f643dcd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:14.744 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.746+0000 7f643dcd27c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:06:14.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.746+0000 7f643dcd27c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:06:14.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.746+0000 7f643dcd27c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:06:14.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:14.782+0000 7f643dcd27c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T08:06:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T08:06:21.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:06:21.192 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:06:21.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:06:21.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:06:21.767 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:06:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:06:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:06:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:06:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:06:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:06:21.802 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:21.802+0000 7f8338b577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:21.837 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:21.838+0000 7f8338b577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:21.839 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:21.838+0000 7f8338b577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:22.231 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:06:22.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:06:22.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:22.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:22.402 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:22.402+0000 7f8338b577c0 -1 Falling back to public interface 2024-04-25T08:06:22.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:23.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:23.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:23.672 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:06:23.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:06:23.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:23.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:24.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:25.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:25.054+0000 7f8338b577c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:06:25.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:25.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:25.144 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:06:25.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:06:25.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:25.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:25.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:26.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:26.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:26.644 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:06:26.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:06:26.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:26.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:27.084 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/904088055,v1:127.0.0.1:6803/904088055] [v2:127.0.0.1:6804/904088055,v1:127.0.0.1:6805/904088055] exists,up 2fc72a75-fcb8-4d2b-976b-cdb000f47c67 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:06:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:27.088 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:06:27.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T08:06:27.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:06:27.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3f75959d-ae0a-41ad-8e5e-a5b6cefc4b40 2024-04-25T08:06:27.090 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 3f75959d-ae0a-41ad-8e5e-a5b6cefc4b40 2024-04-25T08:06:27.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3f75959d-ae0a-41ad-8e5e-a5b6cefc4b40' 2024-04-25T08:06:27.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:06:27.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADDypmJx7kBhAA+G3UCEV0k49C6cmKWoHPOQ== 2024-04-25T08:06:27.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADDypmJx7kBhAA+G3UCEV0k49C6cmKWoHPOQ=="}' 2024-04-25T08:06:27.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3f75959d-ae0a-41ad-8e5e-a5b6cefc4b40 -i td/osd-recovery-prio/1/new.json 2024-04-25T08:06:27.645 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:06:27.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T08:06:27.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADDypmJx7kBhAA+G3UCEV0k49C6cmKWoHPOQ== --osd-uuid 3f75959d-ae0a-41ad-8e5e-a5b6cefc4b40 2024-04-25T08:06:27.699 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.698+0000 7fe27b45c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:27.728 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.730+0000 7fe27b45c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:27.730 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.730+0000 7fe27b45c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:27.731 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.730+0000 7fe27b45c7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:06:27.731 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.730+0000 7fe27b45c7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:06:27.731 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.730+0000 7fe27b45c7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:06:27.762 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:27.762+0000 7fe27b45c7c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T08:06:34.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T08:06:34.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:06:34.648 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:06:34.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:06:34.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:06:35.253 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:06:35.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:06:35.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:06:35.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:06:35.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:06:35.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:06:35.290 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:35.290+0000 7f0e04f7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:35.301 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:35.302+0000 7f0e04f7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:35.302 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:35.302+0000 7f0e04f7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:35.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:06:35.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:06:35.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:06:35.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:06:35.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:06:35.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:35.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:06:35.689 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:06:35.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:35.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:06:35.886 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:35.886+0000 7f0e04f7a7c0 -1 Falling back to public interface 2024-04-25T08:06:36.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:37.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:37.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:37.111 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:06:37.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:06:37.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:37.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:06:37.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:38.535 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:38.534+0000 7f0e04f7a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:06:38.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:38.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:38.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:06:38.543 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:06:38.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:38.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:06:39.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:40.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:40.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:40.075 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:06:40.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:06:40.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:40.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:06:40.506 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4181376236,v1:127.0.0.1:6811/4181376236] [v2:127.0.0.1:6812/4181376236,v1:127.0.0.1:6813/4181376236] exists,up 3f75959d-ae0a-41ad-8e5e-a5b6cefc4b40 2024-04-25T08:06:40.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:06:40.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:06:40.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-04-25T08:06:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:06:40.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:06:40.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:06:40.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9a01346-dbe0-4957-b6a4-daef20c71fa2 2024-04-25T08:06:40.514 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 b9a01346-dbe0-4957-b6a4-daef20c71fa2 2024-04-25T08:06:40.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b9a01346-dbe0-4957-b6a4-daef20c71fa2' 2024-04-25T08:06:40.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:06:40.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQDypmKe8NIBAAS1eNLuncM0bKqRPl5/gkOw== 2024-04-25T08:06:40.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQDypmKe8NIBAAS1eNLuncM0bKqRPl5/gkOw=="}' 2024-04-25T08:06:40.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9a01346-dbe0-4957-b6a4-daef20c71fa2 -i td/osd-recovery-prio/2/new.json 2024-04-25T08:06:41.166 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:06:41.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-04-25T08:06:41.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQDypmKe8NIBAAS1eNLuncM0bKqRPl5/gkOw== --osd-uuid b9a01346-dbe0-4957-b6a4-daef20c71fa2 2024-04-25T08:06:41.228 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.225+0000 7feebab417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:41.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.305+0000 7feebab417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:41.308 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.305+0000 7feebab417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:41.309 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.309+0000 7feebab417c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:06:41.309 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.309+0000 7feebab417c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:06:41.309 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.309+0000 7feebab417c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:06:41.427 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:41.429+0000 7feebab417c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-04-25T08:06:48.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-04-25T08:06:48.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:06:48.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:06:48.290 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:06:48.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:06:48.868 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:06:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:06:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:06:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:06:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:06:48.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:06:48.904 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:48.901+0000 7f7b116127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:48.935 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:48.933+0000 7f7b116127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:48.936 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:48.937+0000 7f7b116127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:06:49.392 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:06:49.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:49.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:49.494 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:49.493+0000 7f7b116127c0 -1 Falling back to public interface 2024-04-25T08:06:49.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:50.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:50.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:50.875 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:06:50.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:06:50.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:50.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:51.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:52.058 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:52.057+0000 7f7b116127c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:06:52.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:52.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:52.315 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:06:52.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:06:52.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:52.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:52.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:53.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:53.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:06:53.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:06:53.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:53.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:54.396 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1953063228,v1:127.0.0.1:6819/1953063228] [v2:127.0.0.1:6820/1953063228,v1:127.0.0.1:6821/1953063228] exists,up b9a01346-dbe0-4957-b6a4-daef20c71fa2 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:06:54.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:06:54.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:06:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:06:54.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:06:54.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:06:54.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-04-25T08:06:54.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:06:54.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9014ace1-9924-440a-9928-b8b1998eee3a 2024-04-25T08:06:54.404 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 9014ace1-9924-440a-9928-b8b1998eee3a 2024-04-25T08:06:54.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9014ace1-9924-440a-9928-b8b1998eee3a' 2024-04-25T08:06:54.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:06:54.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeDypmKY+PGRAALwtN2ZHxdT1qSgMiwA1yCQ== 2024-04-25T08:06:54.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeDypmKY+PGRAALwtN2ZHxdT1qSgMiwA1yCQ=="}' 2024-04-25T08:06:54.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9014ace1-9924-440a-9928-b8b1998eee3a -i td/osd-recovery-prio/3/new.json 2024-04-25T08:06:54.914 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:06:54.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-04-25T08:06:54.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeDypmKY+PGRAALwtN2ZHxdT1qSgMiwA1yCQ== --osd-uuid 9014ace1-9924-440a-9928-b8b1998eee3a 2024-04-25T08:06:54.972 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:54.969+0000 7fda10a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:55.013 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:55.013+0000 7fda10a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:55.014 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:55.013+0000 7fda10a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:55.015 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:55.013+0000 7fda10a9c7c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:06:55.015 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:55.013+0000 7fda10a9c7c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:06:55.015 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:55.013+0000 7fda10a9c7c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:06:55.060 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:06:55.057+0000 7fda10a9c7c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-04-25T08:07:02.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-04-25T08:07:02.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:07:02.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:07:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:07:02.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:07:02.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:07:02.651 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:07:02.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:07:02.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:07:02.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:07:02.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:07:02.687 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:02.685+0000 7f53be0377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:02.709 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:02.709+0000 7f53be0377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:02.711 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:02.709+0000 7f53be0377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:03.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:07:03.161 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:07:03.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:03.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:07:03.282 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:03.281+0000 7f53be0377c0 -1 Falling back to public interface 2024-04-25T08:07:03.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:04.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:04.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:04.610 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:07:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:07:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:07:05.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:06.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:06.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:06.037 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:07:06.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:07:06.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:06.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:07:06.072 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:06.069+0000 7f53be0377c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:07:06.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:07.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:07.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:07.565 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:07:07.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:07:07.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:07.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:07:08.005 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3795429938,v1:127.0.0.1:6827/3795429938] [v2:127.0.0.1:6828/3795429938,v1:127.0.0.1:6829/3795429938] exists,up 9014ace1-9924-440a-9928-b8b1998eee3a 2024-04-25T08:07:08.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:07:08.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:07:08.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:07:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:07:08.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-04-25T08:07:08.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:07:08.012 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 408683a1-00d7-4d6d-af4f-32b629595b19 2024-04-25T08:07:08.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=408683a1-00d7-4d6d-af4f-32b629595b19 2024-04-25T08:07:08.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 408683a1-00d7-4d6d-af4f-32b629595b19' 2024-04-25T08:07:08.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:07:08.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsDypmtQ5NAhAA0qiG60AKdgNMpS51oLxN3A== 2024-04-25T08:07:08.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsDypmtQ5NAhAA0qiG60AKdgNMpS51oLxN3A=="}' 2024-04-25T08:07:08.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 408683a1-00d7-4d6d-af4f-32b629595b19 -i td/osd-recovery-prio/4/new.json 2024-04-25T08:07:08.524 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:07:08.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-04-25T08:07:08.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsDypmtQ5NAhAA0qiG60AKdgNMpS51oLxN3A== --osd-uuid 408683a1-00d7-4d6d-af4f-32b629595b19 2024-04-25T08:07:08.578 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.577+0000 7fa63438b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:08.595 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.593+0000 7fa63438b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:08.597 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.597+0000 7fa63438b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:08.598 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.597+0000 7fa63438b7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:07:08.598 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.597+0000 7fa63438b7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:07:08.598 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.597+0000 7fa63438b7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:07:08.652 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:08.649+0000 7fa63438b7c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-04-25T08:07:15.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-04-25T08:07:15.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:07:15.604 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T08:07:15.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:07:15.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:07:16.179 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T08:07:16.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:07:16.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8b543f91-c8b5-43e6-881e-055ec2b5af0d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:07:16.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:07:16.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:07:16.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:07:16.215 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:16.213+0000 7f2b50b437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:16.246 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:16.245+0000 7f2b50b437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:16.248 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:16.245+0000 7f2b50b437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:16.638 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:07:16.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:07:16.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:16.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:07:16.822 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:16.821+0000 7f2b50b437c0 -1 Falling back to public interface 2024-04-25T08:07:17.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:18.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:18.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:18.088 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:07:18.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:07:18.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:18.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:07:18.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:19.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:19.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:19.554 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:07:19.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:07:19.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:19.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:07:19.645 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:07:19.645+0000 7f2b50b437c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:07:20.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:21.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:21.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:21.043 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:07:21.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:07:21.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:21.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:07:21.507 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1856108082,v1:127.0.0.1:6835/1856108082] [v2:127.0.0.1:6836/1856108082,v1:127.0.0.1:6837/1856108082] exists,up 408683a1-00d7-4d6d-af4f-32b629595b19 2024-04-25T08:07:21.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:07:21.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:07:21.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:07:21.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-04-25T08:07:21.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:21.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-04-25T08:07:21.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:07:22.089 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test1' already exists 2024-04-25T08:07:22.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:23.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-04-25T08:07:24.233 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T08:07:24.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:24.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-04-25T08:07:24.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T08:07:25.153 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test2' already exists 2024-04-25T08:07:25.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-04-25T08:07:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 2 size to 2 2024-04-25T08:07:27.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:27.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-04-25T08:07:27.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T08:07:28.174 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test3' already exists 2024-04-25T08:07:28.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:29.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T08:07:30.180 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 3 size to 2 2024-04-25T08:07:30.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:30.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-04-25T08:07:30.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-04-25T08:07:30.961 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test4' already exists 2024-04-25T08:07:30.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:31.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-04-25T08:07:33.014 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 4 size to 2 2024-04-25T08:07:33.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:33.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-04-25T08:07:33.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-04-25T08:07:33.822 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test5' already exists 2024-04-25T08:07:33.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:34.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-04-25T08:07:36.549 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 5 size to 2 2024-04-25T08:07:36.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:36.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-04-25T08:07:36.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-04-25T08:07:37.244 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test6' already exists 2024-04-25T08:07:37.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:38.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-04-25T08:07:39.119 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 6 size to 2 2024-04-25T08:07:39.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:39.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-04-25T08:07:39.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-04-25T08:07:39.912 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test7' already exists 2024-04-25T08:07:39.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:40.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T08:07:41.870 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 7 size to 2 2024-04-25T08:07:41.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:41.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-04-25T08:07:41.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-04-25T08:07:42.753 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test8' already exists 2024-04-25T08:07:42.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:43.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-04-25T08:07:44.769 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 8 size to 2 2024-04-25T08:07:44.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:44.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-04-25T08:07:44.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-04-25T08:07:45.616 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test9' already exists 2024-04-25T08:07:45.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:46.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-04-25T08:07:47.692 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 9 size to 2 2024-04-25T08:07:47.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:47.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-04-25T08:07:47.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-04-25T08:07:48.607 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test10' already exists 2024-04-25T08:07:48.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-04-25T08:07:50.700 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 10 size to 2 2024-04-25T08:07:50.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-04-25T08:07:55.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-04-25T08:07:55.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:07:55.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:07:55.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:07:55.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:07:55.730 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:07:55.730 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:07:55.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:07:55.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:07:55.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:07:55.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:07:55.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:07:55.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:07:55.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:07:55.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:07:55.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:07:56.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:56.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:07:56.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T08:07:56.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T08:07:56.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T08:07:56.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:56.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:07:56.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T08:07:56.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T08:07:56.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672978' 2024-04-25T08:07:56.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:56.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:07:56.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-04-25T08:07:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-04-25T08:07:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672978 2-64424509456' 2024-04-25T08:07:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:56.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:07:57.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345932 2024-04-25T08:07:57.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345932 2024-04-25T08:07:57.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672978 2-64424509456 3-85899345932' 2024-04-25T08:07:57.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:57.035 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:07:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182410 2024-04-25T08:07:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182410 2024-04-25T08:07:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672978 2-64424509456 3-85899345932 4-107374182410' 2024-04-25T08:07:57.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:57.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T08:07:57.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:57.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:07:57.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T08:07:57.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:57.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T08:07:57.200 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836502 2024-04-25T08:07:57.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T08:07:57.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:07:57.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-25T08:07:57.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:57.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T08:07:57.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:57.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:07:57.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T08:07:57.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:57.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T08:07:57.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T08:07:57.630 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672978 2024-04-25T08:07:57.630 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:07:58.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-25T08:07:58.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:58.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-04-25T08:07:58.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:58.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:07:58.069 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-04-25T08:07:58.069 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:58.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-04-25T08:07:58.071 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509456 2024-04-25T08:07:58.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-25T08:07:58.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:07:58.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509456 2024-04-25T08:07:58.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:58.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345932 2024-04-25T08:07:58.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:58.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:07:58.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345932 2024-04-25T08:07:58.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:58.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345932 2024-04-25T08:07:58.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-04-25T08:07:58.515 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345932 2024-04-25T08:07:58.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:07:58.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345932 -lt 85899345932 2024-04-25T08:07:58.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:58.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182410 2024-04-25T08:07:58.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:58.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:07:58.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182410 2024-04-25T08:07:58.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:58.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182410 2024-04-25T08:07:58.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-04-25T08:07:58.956 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182410 2024-04-25T08:07:58.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:07:59.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182410 -lt 107374182410 2024-04-25T08:07:59.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:07:59.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:59.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:00.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:00.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:00.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-04-25T08:08:00.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:00.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:00.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:01.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-04-25T08:08:01.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:08:01.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:08:01.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:08:01.469 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:08:01.469 INFO:tasks.workunit.client.0.smithi031.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:51.553934+0000 0'0 82:20 [2,1] 2 [2,1] 2 0'0 2024-04-25T08:07:48.188288+0000 0'0 2024-04-25T08:07:48.188288+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:36:07.421311+0000 0 0 2024-04-25T08:08:01.469 INFO:tasks.workunit.client.0.smithi031.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:48.811538+0000 0'0 82:32 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:07:45.316760+0000 0'0 2024-04-25T08:07:45.316760+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:05:33.008740+0000 0 0 2024-04-25T08:08:01.469 INFO:tasks.workunit.client.0.smithi031.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:45.862528+0000 0'0 82:42 [3,4] 3 [3,4] 3 0'0 2024-04-25T08:07:42.392284+0000 0'0 2024-04-25T08:07:42.392284+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:32:40.615857+0000 0 0 2024-04-25T08:08:01.469 INFO:tasks.workunit.client.0.smithi031.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:42.958809+0000 0'0 82:53 [1,4] 1 [1,4] 1 0'0 2024-04-25T08:07:39.591918+0000 0'0 2024-04-25T08:07:39.591918+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:42:01.140779+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:40.260962+0000 0'0 82:52 [4,2] 4 [4,2] 4 0'0 2024-04-25T08:07:36.978798+0000 0'0 2024-04-25T08:07:36.978798+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:48:03.312641+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:37.003022+0000 0'0 82:71 [3,4] 3 [3,4] 3 0'0 2024-04-25T08:07:33.502359+0000 0'0 2024-04-25T08:07:33.502359+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:11:06.553217+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:34.453660+0000 0'0 82:75 [4,0] 4 [4,0] 4 0'0 2024-04-25T08:07:30.775391+0000 0'0 2024-04-25T08:07:30.775391+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:47:09.103545+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:31.161170+0000 0'0 82:84 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:07:27.554022+0000 0'0 2024-04-25T08:07:27.554022+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:04:46.656240+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:28.628837+0000 0'0 82:96 [3,1] 3 [3,1] 3 0'0 2024-04-25T08:07:24.713714+0000 0'0 2024-04-25T08:07:24.713714+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:16:00.752840+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:25.352006+0000 0'0 82:108 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:07:21.890135+0000 0'0 2024-04-25T08:07:21.890135+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:27:18.866670+0000 0 0 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:08:01.470 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-04-25T08:08:01.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-04-25T08:08:01.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-04-25T08:08:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-04-25T08:08:01.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:01.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:08:01.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:01.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-04-25T08:08:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:02.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:02.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T08:08:02.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-04-25T08:08:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:08:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-04-25T08:08:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-04-25T08:08:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-04-25T08:08:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:02.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:02.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:08:02.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:02.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T08:08:02.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-04-25T08:08:02.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:03.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:03.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T08:08:03.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T08:08:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:08:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-04-25T08:08:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-04-25T08:08:03.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:03.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-04-25T08:08:03.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:03.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:03.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T08:08:03.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T08:08:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:08:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-04-25T08:08:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T08:08:03.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:03.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-04-25T08:08:03.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:04.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:04.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T08:08:04.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-04-25T08:08:04.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:08:04.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:04.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:08:04.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:04.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T08:08:04.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-04-25T08:08:04.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T08:08:04.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T08:08:05.088 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test2' does not exist 2024-04-25T08:08:05.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:05.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-04-25T08:08:05.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:05.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-04-25T08:08:05.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-04-25T08:08:05.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-04-25T08:08:05.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-04-25T08:08:05.816 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test4' does not exist 2024-04-25T08:08:05.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:05.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-04-25T08:08:05.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-04-25T08:08:05.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-04-25T08:08:05.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-04-25T08:08:06.562 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test5' does not exist 2024-04-25T08:08:06.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:06.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-04-25T08:08:06.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-04-25T08:08:06.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-04-25T08:08:06.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-04-25T08:08:07.113 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test6' does not exist 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-04-25T08:08:07.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-04-25T08:08:07.721 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test8' does not exist 2024-04-25T08:08:07.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:07.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-04-25T08:08:07.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-04-25T08:08:07.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-04-25T08:08:07.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-04-25T08:08:08.596 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test9' does not exist 2024-04-25T08:08:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-04-25T08:08:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-04-25T08:08:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-04-25T08:08:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-04-25T08:08:09.306 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test10' does not exist 2024-04-25T08:08:09.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-04-25T08:08:10.505 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 3 size to 1 2024-04-25T08:08:10.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-04-25T08:08:11.761 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 7 size to 1 2024-04-25T08:08:11.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-04-25T08:08:11.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:08:11.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:08:11.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:08:11.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:08:11.788 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:08:11.788 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:08:11.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:08:11.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:08:11.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:08:11.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:08:11.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:08:11.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:08:11.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:08:11.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:08:11.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:08:12.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:08:12.387 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:08:12.387 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:08:12.388 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:08:12.388 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:08:12.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:08:12.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:12.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:08:12.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-04-25T08:08:12.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-04-25T08:08:12.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-04-25T08:08:12.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:12.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:08:12.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T08:08:12.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T08:08:12.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672983' 2024-04-25T08:08:12.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:12.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:08:12.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509461 2024-04-25T08:08:12.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509461 2024-04-25T08:08:12.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672983 2-64424509461' 2024-04-25T08:08:12.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:12.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:08:13.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345938 2024-04-25T08:08:13.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345938 2024-04-25T08:08:13.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672983 2-64424509461 3-85899345938' 2024-04-25T08:08:13.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:13.074 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:08:13.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182415 2024-04-25T08:08:13.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182415 2024-04-25T08:08:13.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672983 2-64424509461 3-85899345938 4-107374182415' 2024-04-25T08:08:13.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:13.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-04-25T08:08:13.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:13.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:08:13.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-04-25T08:08:13.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-04-25T08:08:13.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-04-25T08:08:13.235 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836507 2024-04-25T08:08:13.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:08:13.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-04-25T08:08:13.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:13.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T08:08:13.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:13.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:08:13.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T08:08:13.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:13.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T08:08:13.655 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672983 2024-04-25T08:08:13.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T08:08:13.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:08:14.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T08:08:14.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:14.080 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509461 2024-04-25T08:08:14.080 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:14.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:08:14.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509461 2024-04-25T08:08:14.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:14.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509461 2024-04-25T08:08:14.083 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509461 2024-04-25T08:08:14.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-04-25T08:08:14.084 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:08:14.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509461 2024-04-25T08:08:14.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:14.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:14.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345938 2024-04-25T08:08:14.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:08:14.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345938 2024-04-25T08:08:14.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:14.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345938 2024-04-25T08:08:14.511 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345938 2024-04-25T08:08:14.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2024-04-25T08:08:14.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:08:14.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345938 -lt 85899345938 2024-04-25T08:08:14.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:14.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182415 2024-04-25T08:08:14.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:14.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:08:14.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182415 2024-04-25T08:08:14.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:14.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182415 2024-04-25T08:08:14.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-04-25T08:08:14.957 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182415 2024-04-25T08:08:14.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:08:15.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182415 2024-04-25T08:08:15.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:08:15.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:15.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:16.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:08:16.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:16.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:16.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:16.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:08:16.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:08:16.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:08:16.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T08:08:16.981 INFO:tasks.workunit.client.0.smithi031.stderr:10+0 records in 2024-04-25T08:08:16.981 INFO:tasks.workunit.client.0.smithi031.stderr:10+0 records out 2024-04-25T08:08:16.981 INFO:tasks.workunit.client.0.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0389274 s, 269 MB/s 2024-04-25T08:08:16.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-04-25T08:08:16.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:08:16.982 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:08:16.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:16.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T08:08:17.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:17.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T08:08:18.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:18.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T08:08:19.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:19.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T08:08:20.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:20.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T08:08:20.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:20.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T08:08:21.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:21.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T08:08:22.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:22.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T08:08:23.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:23.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T08:08:23.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:23.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T08:08:24.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:24.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T08:08:25.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:25.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T08:08:26.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:26.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T08:08:26.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:26.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T08:08:27.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:27.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T08:08:28.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:28.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T08:08:29.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:29.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T08:08:29.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:29.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T08:08:30.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:30.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T08:08:31.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:31.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T08:08:32.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:32.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T08:08:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T08:08:33.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:33.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T08:08:34.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:34.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T08:08:35.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:35.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T08:08:35.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:35.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T08:08:36.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:36.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T08:08:37.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:37.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T08:08:38.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:38.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T08:08:38.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:38.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T08:08:39.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:39.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T08:08:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:40.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T08:08:41.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:41.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T08:08:42.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:42.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T08:08:42.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:42.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T08:08:43.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:43.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T08:08:44.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:44.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T08:08:45.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:45.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T08:08:46.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:46.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T08:08:46.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:46.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T08:08:47.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:47.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T08:08:48.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:48.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T08:08:49.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:49.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T08:08:49.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:49.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T08:08:50.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:50.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T08:08:51.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:51.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T08:08:52.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:52.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T08:08:52.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:52.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T08:08:53.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:53.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T08:08:54.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:54.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T08:08:55.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:55.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T08:08:56.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:56.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T08:08:56.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:56.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T08:08:57.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:57.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T08:08:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T08:08:59.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:59.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T08:08:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T08:09:00.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:00.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T08:09:01.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:01.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T08:09:02.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:02.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T08:09:02.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:02.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T08:09:03.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:03.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T08:09:04.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:04.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T08:09:05.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:05.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T08:09:06.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:06.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T08:09:06.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:06.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T08:09:07.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:07.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T08:09:08.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:08.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T08:09:09.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:09.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T08:09:10.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:10.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T08:09:11.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:11.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T08:09:11.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:11.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T08:09:12.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:12.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T08:09:13.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:13.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T08:09:14.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:14.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T08:09:14.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:14.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T08:09:15.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:15.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T08:09:16.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:16.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T08:09:17.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:17.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T08:09:17.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:17.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T08:09:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T08:09:19.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:19.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T08:09:20.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:20.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T08:09:20.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:20.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T08:09:21.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:21.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T08:09:22.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:22.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T08:09:23.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:23.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T08:09:23.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:23.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T08:09:24.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:24.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T08:09:25.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:25.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T08:09:26.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:26.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T08:09:26.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:26.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T08:09:27.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:27.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T08:09:28.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:28.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T08:09:28.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:28.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T08:09:29.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:29.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T08:09:30.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:30.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T08:09:31.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:31.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T08:09:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T08:09:32.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:32.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T08:09:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T08:09:34.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:34.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T08:09:34.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:34.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T08:09:35.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:35.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T08:09:36.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:36.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T08:09:37.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:37.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T08:09:37.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:37.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T08:09:38.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:38.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T08:09:39.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:39.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T08:09:40.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:40.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T08:09:40.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:40.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T08:09:41.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:41.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T08:09:42.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:42.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T08:09:43.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:43.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T08:09:43.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:43.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T08:09:44.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:44.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T08:09:45.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:45.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T08:09:46.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:46.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T08:09:46.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:46.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T08:09:47.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:47.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T08:09:48.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:48.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T08:09:49.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:49.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T08:09:49.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:49.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T08:09:50.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:50.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T08:09:51.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:51.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T08:09:52.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:52.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T08:09:52.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:52.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T08:09:53.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:53.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T08:09:54.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:54.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T08:09:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T08:09:56.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:56.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T08:09:56.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:56.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T08:09:57.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:57.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T08:09:58.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:58.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T08:09:59.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:59.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T08:09:59.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:59.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T08:10:00.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:00.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T08:10:01.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:01.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T08:10:02.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:02.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T08:10:02.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:02.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T08:10:03.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:03.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T08:10:04.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:04.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T08:10:05.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:05.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T08:10:05.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:05.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T08:10:06.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:06.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T08:10:07.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:07.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T08:10:08.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:08.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T08:10:08.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:08.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T08:10:09.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:09.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T08:10:10.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:10.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T08:10:11.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:11.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T08:10:12.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:12.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T08:10:13.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:13.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T08:10:13.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:13.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T08:10:14.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:14.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T08:10:15.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:15.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T08:10:16.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:16.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T08:10:16.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:16.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T08:10:17.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:17.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T08:10:18.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:18.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T08:10:19.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:19.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T08:10:20.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:20.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T08:10:20.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:20.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T08:10:21.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:21.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T08:10:22.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:22.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T08:10:23.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:23.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T08:10:23.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:23.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T08:10:24.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:24.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T08:10:25.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:25.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T08:10:26.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:26.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T08:10:26.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:26.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T08:10:27.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:27.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T08:10:28.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:28.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T08:10:29.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:29.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T08:10:29.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:29.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T08:10:30.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:30.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T08:10:31.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:31.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T08:10:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T08:10:33.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:33.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T08:10:33.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:33.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T08:10:34.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:34.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T08:10:35.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T08:10:36.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:36.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T08:10:36.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:36.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T08:10:37.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:37.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T08:10:38.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:38.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T08:10:39.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:39.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T08:10:39.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:39.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T08:10:40.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:40.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T08:10:41.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:41.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T08:10:42.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:42.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T08:10:43.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:43.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T08:10:43.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:43.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T08:10:44.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:44.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T08:10:45.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:45.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T08:10:46.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:46.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T08:10:47.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:47.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T08:10:47.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:47.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T08:10:48.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:48.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T08:10:49.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:49.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T08:10:50.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-04-25T08:10:50.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-04-25T08:10:50.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:10:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:10:50.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:50.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T08:10:50.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:50.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T08:10:51.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:51.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T08:10:51.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:51.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T08:10:52.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:52.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T08:10:52.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:52.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T08:10:52.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:52.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T08:10:53.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:53.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T08:10:53.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:53.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T08:10:54.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:54.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T08:10:54.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:54.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T08:10:55.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:55.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T08:10:55.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:55.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T08:10:56.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:56.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T08:10:56.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:56.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T08:10:57.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:57.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T08:10:57.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:57.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T08:10:58.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:58.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T08:10:58.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:58.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T08:10:58.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:58.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T08:10:59.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:59.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T08:10:59.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:59.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T08:11:00.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:00.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T08:11:00.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:00.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T08:11:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T08:11:01.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:01.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T08:11:02.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:02.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T08:11:02.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:02.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T08:11:03.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:03.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T08:11:03.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:03.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T08:11:03.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:03.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T08:11:04.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:04.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T08:11:04.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:04.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T08:11:05.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:05.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T08:11:05.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:05.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T08:11:06.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:06.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T08:11:06.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:06.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T08:11:07.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:07.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T08:11:07.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:07.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T08:11:08.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:08.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T08:11:08.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:08.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T08:11:08.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:08.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T08:11:09.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:09.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T08:11:09.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:09.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T08:11:10.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:10.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T08:11:10.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:10.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T08:11:11.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:11.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T08:11:11.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:11.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T08:11:12.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:12.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T08:11:12.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:12.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T08:11:13.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:13.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T08:11:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:13.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T08:11:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T08:11:14.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:14.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T08:11:14.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:14.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T08:11:15.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:15.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T08:11:15.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:15.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T08:11:16.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:16.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T08:11:16.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:16.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T08:11:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:17.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T08:11:17.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:17.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T08:11:18.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:18.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T08:11:18.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:18.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T08:11:19.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:19.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T08:11:19.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:19.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T08:11:19.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:19.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T08:11:20.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:20.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T08:11:20.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:20.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T08:11:21.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:21.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T08:11:21.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:21.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T08:11:22.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:22.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T08:11:22.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:22.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T08:11:23.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:23.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T08:11:23.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:23.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T08:11:24.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:24.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T08:11:24.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:24.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T08:11:25.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:25.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T08:11:25.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:25.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T08:11:26.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:26.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T08:11:26.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:26.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T08:11:26.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:26.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T08:11:27.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:27.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T08:11:27.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:27.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T08:11:28.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:28.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T08:11:28.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:28.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T08:11:29.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:29.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T08:11:29.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:29.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T08:11:30.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:30.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T08:11:30.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:30.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T08:11:31.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:31.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T08:11:31.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:31.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T08:11:31.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:31.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T08:11:32.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:32.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T08:11:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:32.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T08:11:33.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:33.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T08:11:33.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:33.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T08:11:34.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:34.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T08:11:34.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:34.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T08:11:35.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:35.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T08:11:35.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:35.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T08:11:35.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:35.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T08:11:36.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:36.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T08:11:36.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:36.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T08:11:37.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:37.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T08:11:37.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:37.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T08:11:38.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:38.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T08:11:38.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:38.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T08:11:39.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:39.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T08:11:39.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:39.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T08:11:40.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:40.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T08:11:40.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:40.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T08:11:40.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:40.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T08:11:41.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:41.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T08:11:41.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:41.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T08:11:42.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:42.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T08:11:42.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:42.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T08:11:43.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:43.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T08:11:43.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:43.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T08:11:44.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:44.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T08:11:44.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:44.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T08:11:45.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:45.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T08:11:45.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:45.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T08:11:46.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:46.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T08:11:46.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:46.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T08:11:46.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:46.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T08:11:47.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:47.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T08:11:47.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:47.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T08:11:48.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:48.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T08:11:48.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:48.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T08:11:49.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:49.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T08:11:49.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:49.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T08:11:50.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:50.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T08:11:50.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:50.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T08:11:50.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:50.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T08:11:51.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:51.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T08:11:51.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:51.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T08:11:52.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:52.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T08:11:52.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:52.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T08:11:53.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:53.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T08:11:53.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:53.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T08:11:54.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:54.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T08:11:54.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:54.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T08:11:55.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:55.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T08:11:55.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:55.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T08:11:56.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:56.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T08:11:56.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:56.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T08:11:56.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:56.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T08:11:57.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:57.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T08:11:57.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:57.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T08:11:58.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:58.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T08:11:58.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:58.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T08:11:59.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:59.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T08:11:59.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:59.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T08:12:00.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:00.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T08:12:00.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:00.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T08:12:01.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:01.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T08:12:01.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:01.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T08:12:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:01.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T08:12:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T08:12:02.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:02.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T08:12:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T08:12:03.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:03.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T08:12:04.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:04.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T08:12:04.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:04.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T08:12:05.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:05.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T08:12:05.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:05.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T08:12:06.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:06.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T08:12:06.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:06.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T08:12:06.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:06.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T08:12:07.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:07.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T08:12:07.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:07.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T08:12:08.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:08.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T08:12:08.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:08.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T08:12:09.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:09.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T08:12:09.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:09.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T08:12:10.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:10.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T08:12:10.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:10.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T08:12:10.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:10.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T08:12:11.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:11.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T08:12:11.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:11.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T08:12:12.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:12.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T08:12:12.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:12.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T08:12:13.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:13.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T08:12:13.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:13.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T08:12:14.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:14.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T08:12:14.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:14.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T08:12:15.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:15.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T08:12:15.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:15.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T08:12:16.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:16.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T08:12:16.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:16.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T08:12:17.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:17.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T08:12:17.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:17.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T08:12:18.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:18.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T08:12:18.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:18.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T08:12:18.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:18.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T08:12:19.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:19.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T08:12:19.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:19.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T08:12:20.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:20.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T08:12:20.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:20.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T08:12:21.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:21.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T08:12:21.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-04-25T08:12:21.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-04-25T08:12:21.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:12:21.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:12:21.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:21.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-04-25T08:12:22.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:22.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-04-25T08:12:22.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:22.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-04-25T08:12:23.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:23.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-04-25T08:12:23.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:23.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-04-25T08:12:23.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:23.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-04-25T08:12:24.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:24.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-04-25T08:12:24.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:24.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-04-25T08:12:25.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:25.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-04-25T08:12:25.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:25.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-04-25T08:12:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-04-25T08:12:26.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:26.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-04-25T08:12:27.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:27.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-04-25T08:12:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:27.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-04-25T08:12:27.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:27.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-04-25T08:12:28.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:28.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-04-25T08:12:28.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:28.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-04-25T08:12:29.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:29.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-04-25T08:12:29.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:29.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-04-25T08:12:30.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:30.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-04-25T08:12:30.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:30.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-04-25T08:12:31.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:31.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-04-25T08:12:31.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:31.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-04-25T08:12:32.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:32.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-04-25T08:12:32.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:32.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-04-25T08:12:32.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:32.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-04-25T08:12:33.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:33.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-04-25T08:12:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-04-25T08:12:34.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:34.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-04-25T08:12:34.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:34.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-04-25T08:12:35.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:35.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-04-25T08:12:35.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:35.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-04-25T08:12:36.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:36.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-04-25T08:12:36.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:36.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-04-25T08:12:36.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:36.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-04-25T08:12:37.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:37.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-04-25T08:12:37.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:37.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-04-25T08:12:38.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:38.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-04-25T08:12:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-04-25T08:12:39.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:39.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-04-25T08:12:39.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:39.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-04-25T08:12:40.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:40.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-04-25T08:12:40.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:40.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-04-25T08:12:41.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:41.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-04-25T08:12:41.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:41.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-04-25T08:12:42.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:42.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-04-25T08:12:42.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:42.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-04-25T08:12:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:43.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-04-25T08:12:43.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:43.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-04-25T08:12:43.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:43.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-04-25T08:12:44.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:44.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-04-25T08:12:44.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:44.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-04-25T08:12:45.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:45.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-04-25T08:12:45.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:45.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-04-25T08:12:46.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:46.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-04-25T08:12:46.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:46.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-04-25T08:12:47.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:47.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-04-25T08:12:47.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:47.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-04-25T08:12:48.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-04-25T08:12:48.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:48.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-04-25T08:12:48.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:48.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-04-25T08:12:49.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:49.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-04-25T08:12:49.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:49.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-04-25T08:12:50.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:50.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-04-25T08:12:50.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:50.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-04-25T08:12:51.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:51.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-04-25T08:12:51.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:51.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-04-25T08:12:52.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:52.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-04-25T08:12:52.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:52.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-04-25T08:12:53.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:53.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-04-25T08:12:53.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:53.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-04-25T08:12:54.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:54.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-04-25T08:12:54.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:54.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-04-25T08:12:54.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:54.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-04-25T08:12:55.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:55.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-04-25T08:12:55.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:55.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-04-25T08:12:56.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:56.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-04-25T08:12:56.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:56.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-04-25T08:12:57.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:57.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-04-25T08:12:57.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:57.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-04-25T08:12:58.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:58.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-04-25T08:12:58.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:58.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-04-25T08:12:58.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:58.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-04-25T08:12:59.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:59.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-04-25T08:12:59.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:59.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-04-25T08:13:00.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:00.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-04-25T08:13:00.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:00.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-04-25T08:13:01.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:01.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-04-25T08:13:01.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:01.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-04-25T08:13:02.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:02.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-04-25T08:13:02.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:02.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-04-25T08:13:02.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:02.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-04-25T08:13:03.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:03.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-04-25T08:13:03.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:03.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-04-25T08:13:04.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:04.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-04-25T08:13:04.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:04.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-04-25T08:13:05.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:05.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-04-25T08:13:05.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:05.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-04-25T08:13:06.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:06.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-04-25T08:13:06.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:06.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-04-25T08:13:06.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:06.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-04-25T08:13:07.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:07.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-04-25T08:13:07.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:07.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-04-25T08:13:08.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:08.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-04-25T08:13:08.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:08.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-04-25T08:13:09.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:09.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-04-25T08:13:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-04-25T08:13:10.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:10.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-04-25T08:13:10.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:10.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-04-25T08:13:10.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:10.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-04-25T08:13:11.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:11.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-04-25T08:13:11.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:11.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-04-25T08:13:12.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:12.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-04-25T08:13:12.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:12.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-04-25T08:13:13.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:13.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-04-25T08:13:13.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:13.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-04-25T08:13:14.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:14.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-04-25T08:13:14.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:14.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-04-25T08:13:15.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:15.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-04-25T08:13:15.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:15.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-04-25T08:13:16.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:16.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-04-25T08:13:16.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:16.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-04-25T08:13:17.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:17.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-04-25T08:13:17.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:17.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-04-25T08:13:18.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:18.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-04-25T08:13:18.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:18.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-04-25T08:13:18.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:18.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-04-25T08:13:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-04-25T08:13:19.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:19.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-04-25T08:13:20.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:20.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-04-25T08:13:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:20.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-04-25T08:13:21.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:21.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-04-25T08:13:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:21.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-04-25T08:13:22.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:22.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-04-25T08:13:22.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:22.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-04-25T08:13:22.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:22.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-04-25T08:13:23.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:23.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-04-25T08:13:23.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:23.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-04-25T08:13:24.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:24.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-04-25T08:13:24.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:24.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-04-25T08:13:25.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:25.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-04-25T08:13:25.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:25.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-04-25T08:13:26.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:26.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-04-25T08:13:26.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:26.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-04-25T08:13:27.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:27.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-04-25T08:13:27.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:27.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-04-25T08:13:27.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:27.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-04-25T08:13:28.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:28.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-04-25T08:13:28.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:28.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-04-25T08:13:29.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:29.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-04-25T08:13:29.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:29.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-04-25T08:13:30.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:30.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-04-25T08:13:30.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:30.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-04-25T08:13:31.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:31.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-04-25T08:13:31.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:31.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-04-25T08:13:32.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:32.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-04-25T08:13:32.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:32.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-04-25T08:13:32.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:32.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-04-25T08:13:33.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:33.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-04-25T08:13:33.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:33.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-04-25T08:13:34.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:34.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-04-25T08:13:34.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:34.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-04-25T08:13:35.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:35.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-04-25T08:13:35.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:35.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-04-25T08:13:36.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:36.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-04-25T08:13:36.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:36.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-04-25T08:13:36.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:36.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-04-25T08:13:37.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:37.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-04-25T08:13:37.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:37.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-04-25T08:13:38.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:38.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-04-25T08:13:38.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:38.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-04-25T08:13:39.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:39.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-04-25T08:13:39.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:39.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-04-25T08:13:40.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:40.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-04-25T08:13:40.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:40.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-04-25T08:13:41.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:41.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-04-25T08:13:41.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-04-25T08:13:42.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:42.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-04-25T08:13:42.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:42.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-04-25T08:13:43.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:43.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-04-25T08:13:43.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:43.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-04-25T08:13:44.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:44.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-04-25T08:13:44.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:44.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-04-25T08:13:45.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:45.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-04-25T08:13:45.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:45.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-04-25T08:13:46.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:46.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-04-25T08:13:46.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:46.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-04-25T08:13:47.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:47.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-04-25T08:13:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:47.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-04-25T08:13:48.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:48.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-04-25T08:13:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-04-25T08:13:48.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:48.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-04-25T08:13:49.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:49.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-04-25T08:13:49.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:49.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-04-25T08:13:50.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:50.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-04-25T08:13:50.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:50.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-04-25T08:13:51.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:51.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-04-25T08:13:51.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:51.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-04-25T08:13:52.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:52.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-04-25T08:13:52.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:52.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-04-25T08:13:53.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-04-25T08:13:53.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-04-25T08:13:53.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-04-25T08:13:53.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T08:13:53.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T08:13:53.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T08:13:53.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T08:13:53.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T08:13:53.077 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:13:53.077 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:13:53.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:13:53.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:13:53.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:13:54.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-04-25T08:13:54.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stdout:7.0 194 0 0 0 0 2032140288 0 0 581 0 581 active+clean 2024-04-25T08:08:12.342357+0000 95'581 95:662 [1] 1 [1] 1 0'0 2024-04-25T08:07:39.591918+0000 0'0 2024-04-25T08:07:39.591918+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:17:03.054807+0000 0 0 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:08:11.028274+0000 95'600 95:712 [1] 1 [1] 1 0'0 2024-04-25T08:07:27.554022+0000 0'0 2024-04-25T08:07:27.554022+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:36:12.407401+0000 0 0 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:07:25.352006+0000 95'600 95:733 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:07:21.890135+0000 0'0 2024-04-25T08:07:21.890135+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:27:18.866670+0000 0 0 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:13:54.420 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:13:54.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-04-25T08:13:54.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-04-25T08:13:55.223 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:13:55.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-04-25T08:13:56.060 INFO:tasks.workunit.client.0.smithi031.stderr:noout is set 2024-04-25T08:13:56.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T08:13:57.022 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 7 size to 2 2024-04-25T08:13:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-04-25T08:13:59.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:13:59.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:13:59.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:13:59.053 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:13:59.053 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:13:59.053 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:13:59.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:13:59.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:13:59.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:13:59.175 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:13:59.176 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:13:59.176 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:13:59.176 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:13:59.176 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:13:59.176 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:13:59.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-04-25T08:13:59.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:13:59.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:13:59.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:13:59.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-04-25T08:13:59.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T08:14:01.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:14:01.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:14:01.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:14:02.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-04-25T08:14:02.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:14:02.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:14:02.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:14:02.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:02.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:14:02.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836579 2024-04-25T08:14:02.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836579 2024-04-25T08:14:02.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836579' 2024-04-25T08:14:02.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:02.708 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:14:02.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673055 2024-04-25T08:14:02.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673055 2024-04-25T08:14:02.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836579 1-42949673055' 2024-04-25T08:14:02.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:02.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:14:03.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509533 2024-04-25T08:14:03.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509533 2024-04-25T08:14:03.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509533' 2024-04-25T08:14:03.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:03.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:14:03.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346010 2024-04-25T08:14:03.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346010 2024-04-25T08:14:03.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509533 3-85899346010' 2024-04-25T08:14:03.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:03.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:14:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182487 2024-04-25T08:14:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182487 2024-04-25T08:14:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509533 3-85899346010 4-107374182487' 2024-04-25T08:14:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:03.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:03.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836579 2024-04-25T08:14:03.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:14:03.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836579 2024-04-25T08:14:03.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:03.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836579 2024-04-25T08:14:03.347 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836579 2024-04-25T08:14:03.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-04-25T08:14:03.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:03.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836579 -lt 21474836579 2024-04-25T08:14:03.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:03.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673055 2024-04-25T08:14:03.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:03.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:14:03.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673055 2024-04-25T08:14:03.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:03.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673055 2024-04-25T08:14:03.799 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673055 2024-04-25T08:14:03.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-04-25T08:14:03.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:04.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673055 -lt 42949673055 2024-04-25T08:14:04.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:04.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509533 2024-04-25T08:14:04.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:04.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:14:04.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509533 2024-04-25T08:14:04.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:04.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509533 2024-04-25T08:14:04.232 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509533 2024-04-25T08:14:04.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-04-25T08:14:04.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:14:04.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509533 -lt 64424509533 2024-04-25T08:14:04.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:04.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:04.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346010 2024-04-25T08:14:04.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:14:04.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346010 2024-04-25T08:14:04.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:04.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346010 2024-04-25T08:14:04.687 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899346010 2024-04-25T08:14:04.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346010' 2024-04-25T08:14:04.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:14:05.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346010 -lt 85899346010 2024-04-25T08:14:05.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:05.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182487 2024-04-25T08:14:05.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:05.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:14:05.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182487 2024-04-25T08:14:05.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:05.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182487 2024-04-25T08:14:05.114 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182487 2024-04-25T08:14:05.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-04-25T08:14:05.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:14:05.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182487 -lt 107374182487 2024-04-25T08:14:05.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:05.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:05.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:05.583 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:05.583 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:05.583 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:05.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:05.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:05.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:05.706 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:05.707 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:05.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-04-25T08:14:06.250 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T08:14:06.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-04-25T08:14:08.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:14:08.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:14:08.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:14:08.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:08.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:14:08.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836582 2024-04-25T08:14:08.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836582 2024-04-25T08:14:08.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582' 2024-04-25T08:14:08.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:08.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:14:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673058 2024-04-25T08:14:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673058 2024-04-25T08:14:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673058' 2024-04-25T08:14:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:09.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:14:09.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509537 2024-04-25T08:14:09.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509537 2024-04-25T08:14:09.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673058 2-64424509537' 2024-04-25T08:14:09.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:09.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:14:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346013 2024-04-25T08:14:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346013 2024-04-25T08:14:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673058 2-64424509537 3-85899346013' 2024-04-25T08:14:09.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:09.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:14:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182490 2024-04-25T08:14:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182490 2024-04-25T08:14:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836582 1-42949673058 2-64424509537 3-85899346013 4-107374182490' 2024-04-25T08:14:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:09.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:09.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836582 2024-04-25T08:14:09.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:14:09.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836582 2024-04-25T08:14:09.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:09.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836582 2024-04-25T08:14:09.505 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836582 2024-04-25T08:14:09.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-04-25T08:14:09.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:09.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836582 -lt 21474836582 2024-04-25T08:14:09.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:09.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673058 2024-04-25T08:14:09.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:09.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:14:09.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673058 2024-04-25T08:14:09.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:09.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673058 2024-04-25T08:14:09.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-04-25T08:14:09.968 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673058 2024-04-25T08:14:09.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673058 -lt 42949673058 2024-04-25T08:14:10.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:10.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509537 2024-04-25T08:14:10.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:10.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:14:10.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509537 2024-04-25T08:14:10.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:10.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509537 2024-04-25T08:14:10.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-04-25T08:14:10.425 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509537 2024-04-25T08:14:10.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:14:10.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509537 -lt 64424509537 2024-04-25T08:14:10.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:10.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346013 2024-04-25T08:14:10.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:10.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:14:10.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346013 2024-04-25T08:14:10.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:10.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346013 2024-04-25T08:14:10.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346013' 2024-04-25T08:14:10.907 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899346013 2024-04-25T08:14:10.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:14:11.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346013 -lt 85899346013 2024-04-25T08:14:11.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:11.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:11.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182490 2024-04-25T08:14:11.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:14:11.349 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182490 2024-04-25T08:14:11.349 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:11.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182490 2024-04-25T08:14:11.351 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182490 2024-04-25T08:14:11.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-04-25T08:14:11.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:14:11.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182490 -lt 107374182490 2024-04-25T08:14:11.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:11.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:11.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:11.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:11.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:11.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:11.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:11.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:11.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:11.879 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:11.879 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:11.879 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:11.879 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:11.879 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:11.879 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:11.881 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:11.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:14:12.295 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:14:12.295 INFO:tasks.workunit.client.0.smithi031.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T08:14:01.936994+0000 95'600 107:705 [1,4] 1 [1] 1 0'0 2024-04-25T08:07:39.591918+0000 0'0 2024-04-25T08:07:39.591918+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:14:12.295 INFO:tasks.workunit.client.0.smithi031.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:08:11.028274+0000 95'600 107:733 [1] 1 [1] 1 0'0 2024-04-25T08:07:27.554022+0000 0'0 2024-04-25T08:07:27.554022+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:36:12.407401+0000 0 0 2024-04-25T08:14:12.295 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:14:08.167626+0000 95'600 107:755 [1,2] 1 [1] 1 0'0 2024-04-25T08:07:21.890135+0000 0'0 2024-04-25T08:07:21.890135+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:14:12.295 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:14:12.295 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:14:12.296 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:14:12.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T08:14:13.132 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 3 size to 2 2024-04-25T08:14:13.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-04-25T08:14:15.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:14:15.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:14:15.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:14:15.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:15.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:14:15.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836585 2024-04-25T08:14:15.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836585 2024-04-25T08:14:15.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585' 2024-04-25T08:14:15.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:15.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:14:15.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673062 2024-04-25T08:14:15.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673062 2024-04-25T08:14:15.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062' 2024-04-25T08:14:15.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:15.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:14:16.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509540 2024-04-25T08:14:16.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509540 2024-04-25T08:14:16.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509540' 2024-04-25T08:14:16.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:16.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:14:16.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346016 2024-04-25T08:14:16.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346016 2024-04-25T08:14:16.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509540 3-85899346016' 2024-04-25T08:14:16.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:16.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:14:16.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182494 2024-04-25T08:14:16.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182494 2024-04-25T08:14:16.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509540 3-85899346016 4-107374182494' 2024-04-25T08:14:16.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:16.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836585 2024-04-25T08:14:16.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:16.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:14:16.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836585 2024-04-25T08:14:16.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:16.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836585 2024-04-25T08:14:16.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-04-25T08:14:16.426 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836585 2024-04-25T08:14:16.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:16.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836583 -lt 21474836585 2024-04-25T08:14:16.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:14:17.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:14:17.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:18.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836586 -lt 21474836585 2024-04-25T08:14:18.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:18.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673062 2024-04-25T08:14:18.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:14:18.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673062 2024-04-25T08:14:18.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:18.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673062 2024-04-25T08:14:18.302 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673062 2024-04-25T08:14:18.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-04-25T08:14:18.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:18.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673062 -lt 42949673062 2024-04-25T08:14:18.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:18.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509540 2024-04-25T08:14:18.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:18.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:14:18.781 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:18.781 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509540 2024-04-25T08:14:18.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509540 2024-04-25T08:14:18.782 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509540 2024-04-25T08:14:18.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-04-25T08:14:18.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:14:19.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509540 -lt 64424509540 2024-04-25T08:14:19.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:19.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346016 2024-04-25T08:14:19.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:19.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:14:19.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346016 2024-04-25T08:14:19.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:19.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346016 2024-04-25T08:14:19.251 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899346016 2024-04-25T08:14:19.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346016' 2024-04-25T08:14:19.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:14:19.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346017 -lt 85899346016 2024-04-25T08:14:19.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:19.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182494 2024-04-25T08:14:19.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:19.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:14:19.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182494 2024-04-25T08:14:19.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:19.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182494 2024-04-25T08:14:19.683 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182494 2024-04-25T08:14:19.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-04-25T08:14:19.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:14:20.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182494 -lt 107374182494 2024-04-25T08:14:20.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:20.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:20.125 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:20.125 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:20.125 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:20.125 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:20.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:20.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:20.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:20.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:20.236 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:20.236 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:20.237 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:20.238 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:20.239 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:20.239 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:20.239 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:20.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:14:20.643 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:14:20.643 INFO:tasks.workunit.client.0.smithi031.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T08:14:01.936994+0000 95'600 111:712 [1,4] 1 [1] 1 0'0 2024-04-25T08:07:39.591918+0000 0'0 2024-04-25T08:07:39.591918+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:14:20.643 INFO:tasks.workunit.client.0.smithi031.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:14:14.371475+0000 95'600 111:741 [1,2] 1 [1] 1 0'0 2024-04-25T08:07:27.554022+0000 0'0 2024-04-25T08:07:27.554022+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:14:20.643 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:14:08.167626+0000 95'600 111:762 [1,2] 1 [1] 1 0'0 2024-04-25T08:07:21.890135+0000 0'0 2024-04-25T08:07:21.890135+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:14:20.644 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:14:20.644 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:14:20.644 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:14:20.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-04-25T08:14:20.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:20.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-04-25T08:14:20.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:14:20.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:20.698 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:14:20.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T08:14:20.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-04-25T08:14:20.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T08:14:20.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:20.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:14:20.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-04-25T08:14:20.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:14:20.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-04-25T08:14:20.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:14:20.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-04-25T08:14:20.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:14:21.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-04-25T08:14:21.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-04-25T08:14:23.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:23.206 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:23.206 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:23.206 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:23.207 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:23.207 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:23.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:23.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:23.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:23.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 255, 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:23.345 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:23.346 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:23.346 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:23.346 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:23.346 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:23.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:23.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T08:14:23.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-04-25T08:14:23.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:14:23.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:14:23.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:14:23.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:14:23.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:14:23.815 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:14:23.815 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:14:23.816 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:14:23.816 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:14:23.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:14:23.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:23.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:14:24.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836589 2024-04-25T08:14:24.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836589 2024-04-25T08:14:24.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589' 2024-04-25T08:14:24.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:24.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:14:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673065 2024-04-25T08:14:24.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673065 2024-04-25T08:14:24.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673065' 2024-04-25T08:14:24.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:24.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:14:24.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509544 2024-04-25T08:14:24.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509544 2024-04-25T08:14:24.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673065 2-64424509544' 2024-04-25T08:14:24.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:24.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:14:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346020 2024-04-25T08:14:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346020 2024-04-25T08:14:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673065 2-64424509544 3-85899346020' 2024-04-25T08:14:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:24.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:14:24.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182497 2024-04-25T08:14:24.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182497 2024-04-25T08:14:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673065 2-64424509544 3-85899346020 4-107374182497' 2024-04-25T08:14:24.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:24.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836589 2024-04-25T08:14:24.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:24.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:14:24.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836589 2024-04-25T08:14:24.641 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:24.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836589 2024-04-25T08:14:24.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-04-25T08:14:24.643 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836589 2024-04-25T08:14:24.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:25.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836587 -lt 21474836589 2024-04-25T08:14:25.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:14:26.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:14:26.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:26.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836589 -lt 21474836589 2024-04-25T08:14:26.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:26.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673065 2024-04-25T08:14:26.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:26.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:14:26.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673065 2024-04-25T08:14:26.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:26.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673065 2024-04-25T08:14:26.550 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673065 2024-04-25T08:14:26.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-04-25T08:14:26.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673065 -lt 42949673065 2024-04-25T08:14:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:26.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509544 2024-04-25T08:14:26.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:26.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:14:26.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509544 2024-04-25T08:14:26.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:26.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509544 2024-04-25T08:14:26.996 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509544 2024-04-25T08:14:26.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-04-25T08:14:26.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:14:27.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509544 -lt 64424509544 2024-04-25T08:14:27.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:27.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346020 2024-04-25T08:14:27.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:27.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:14:27.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346020 2024-04-25T08:14:27.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:27.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346020 2024-04-25T08:14:27.429 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899346020 2024-04-25T08:14:27.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346020' 2024-04-25T08:14:27.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:14:27.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346020 -lt 85899346020 2024-04-25T08:14:27.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:27.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182497 2024-04-25T08:14:27.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:27.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:14:27.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182497 2024-04-25T08:14:27.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:27.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182497 2024-04-25T08:14:27.870 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182497 2024-04-25T08:14:27.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182497' 2024-04-25T08:14:27.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:14:28.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182498 -lt 107374182497 2024-04-25T08:14:28.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-04-25T08:14:28.728 INFO:tasks.workunit.client.0.smithi031.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-04-25T08:14:28.728 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T08:14:28.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-04-25T08:14:30.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:30.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:30.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:30.747 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:30.748 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:30.748 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:30.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:30.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:30.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:30.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:30.882 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:30.883 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:30.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-04-25T08:14:30.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-04-25T08:14:30.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:14:30.922 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:30.922 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:14:30.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-04-25T08:14:30.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-04-25T08:14:30.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-04-25T08:14:30.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:30.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:14:30.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-04-25T08:14:30.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:14:30.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-04-25T08:14:31.392 INFO:tasks.workunit.client.0.smithi031.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-04-25T08:14:31.392 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T08:14:31.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-04-25T08:14:36.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:36.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:36.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:36.411 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:36.412 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:36.412 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:36.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:36.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:36.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:36.531 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:36.532 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:36.533 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:36.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:14:36.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:14:36.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:14:36.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:36.980 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:14:37.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836594 2024-04-25T08:14:37.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836594 2024-04-25T08:14:37.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594' 2024-04-25T08:14:37.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:37.157 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:14:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673070 2024-04-25T08:14:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673070 2024-04-25T08:14:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673070' 2024-04-25T08:14:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:37.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:14:37.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509548 2024-04-25T08:14:37.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509548 2024-04-25T08:14:37.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673070 2-64424509548' 2024-04-25T08:14:37.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:37.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:14:37.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346025 2024-04-25T08:14:37.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346025 2024-04-25T08:14:37.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673070 2-64424509548 3-85899346025' 2024-04-25T08:14:37.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:37.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:14:37.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182502 2024-04-25T08:14:37.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182502 2024-04-25T08:14:37.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836594 1-42949673070 2-64424509548 3-85899346025 4-107374182502' 2024-04-25T08:14:37.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:37.840 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836594 2024-04-25T08:14:37.840 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:37.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:14:37.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836594 2024-04-25T08:14:37.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:37.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836594 2024-04-25T08:14:37.843 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836594 2024-04-25T08:14:37.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2024-04-25T08:14:37.844 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:38.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836594 -lt 21474836594 2024-04-25T08:14:38.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:38.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:38.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673070 2024-04-25T08:14:38.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:14:38.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673070 2024-04-25T08:14:38.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:38.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673070 2024-04-25T08:14:38.278 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673070 2024-04-25T08:14:38.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-04-25T08:14:38.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673068 -lt 42949673070 2024-04-25T08:14:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:14:39.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:14:39.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:40.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673070 -lt 42949673070 2024-04-25T08:14:40.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:40.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:40.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509548 2024-04-25T08:14:40.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509548 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:40.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509548 2024-04-25T08:14:40.174 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509548 2024-04-25T08:14:40.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-04-25T08:14:40.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:14:40.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509549 -lt 64424509548 2024-04-25T08:14:40.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:40.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346025 2024-04-25T08:14:40.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:40.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:14:40.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346025 2024-04-25T08:14:40.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:40.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346025 2024-04-25T08:14:40.601 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899346025 2024-04-25T08:14:40.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346025' 2024-04-25T08:14:40.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:14:41.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346025 -lt 85899346025 2024-04-25T08:14:41.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:41.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:41.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182502 2024-04-25T08:14:41.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:14:41.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182502 2024-04-25T08:14:41.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:41.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182502 2024-04-25T08:14:41.034 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182502 2024-04-25T08:14:41.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182502' 2024-04-25T08:14:41.035 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:14:41.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182502 -lt 107374182502 2024-04-25T08:14:41.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:14:41.910 INFO:tasks.workunit.client.0.smithi031.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-04-25T08:14:41.911 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T08:14:41.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-04-25T08:14:43.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:43.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:43.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:43.930 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:43.930 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:43.930 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:43.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:43.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:14:43.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:44.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:44.064 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:44.064 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:44.064 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:44.064 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:44.065 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:44.066 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:44.067 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:44.067 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:44.067 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:44.067 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:44.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:44.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T08:14:44.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-04-25T08:14:44.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:14:44.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:14:44.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:44.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T08:14:44.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-04-25T08:14:44.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T08:14:44.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:14:44.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:14:44.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-04-25T08:14:44.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:14:44.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-04-25T08:14:44.866 INFO:tasks.workunit.client.0.smithi031.stderr:noout is unset 2024-04-25T08:14:44.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-04-25T08:14:45.813 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:14:45.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:14:45.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:14:45.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:14:45.835 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:45.835 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:45.835 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:14:45.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:14:45.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T08:14:45.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T08:14:45.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:14:45.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:14:45.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:14:45.838 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:14:45.838 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:14:45.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:14:45.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:14:45.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:14:45.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:14:45.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:14:45.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:14:45.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:14:45.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:14:45.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:14:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:14:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:14:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:14:46.418 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:14:46.418 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:14:46.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:14:46.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:46.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:14:46.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836598 2024-04-25T08:14:46.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836598 2024-04-25T08:14:46.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836598' 2024-04-25T08:14:46.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:46.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:14:46.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673074 2024-04-25T08:14:46.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673074 2024-04-25T08:14:46.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836598 1-42949673074' 2024-04-25T08:14:46.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:46.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:14:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509552 2024-04-25T08:14:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509552 2024-04-25T08:14:46.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836598 1-42949673074 2-64424509552' 2024-04-25T08:14:46.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:46.913 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:14:47.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346028 2024-04-25T08:14:47.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346028 2024-04-25T08:14:47.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836598 1-42949673074 2-64424509552 3-85899346028' 2024-04-25T08:14:47.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:14:47.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:14:47.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182506 2024-04-25T08:14:47.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182506 2024-04-25T08:14:47.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836598 1-42949673074 2-64424509552 3-85899346028 4-107374182506' 2024-04-25T08:14:47.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:47.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836598 2024-04-25T08:14:47.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:47.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:14:47.239 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836598 2024-04-25T08:14:47.239 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:47.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836598 2024-04-25T08:14:47.240 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836598 2024-04-25T08:14:47.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-04-25T08:14:47.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:14:47.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836598 -lt 21474836598 2024-04-25T08:14:47.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:47.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673074 2024-04-25T08:14:47.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:47.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:14:47.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673074 2024-04-25T08:14:47.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:47.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673074 2024-04-25T08:14:47.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-04-25T08:14:47.789 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673074 2024-04-25T08:14:47.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:14:48.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673074 -lt 42949673074 2024-04-25T08:14:48.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:48.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509552 2024-04-25T08:14:48.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:48.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:14:48.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509552 2024-04-25T08:14:48.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:48.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509552 2024-04-25T08:14:48.300 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509552 2024-04-25T08:14:48.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509552' 2024-04-25T08:14:48.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:14:48.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509552 -lt 64424509552 2024-04-25T08:14:48.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:48.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346028 2024-04-25T08:14:48.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:48.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:14:48.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346028 2024-04-25T08:14:48.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:48.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346028 2024-04-25T08:14:48.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346028' 2024-04-25T08:14:48.836 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899346028 2024-04-25T08:14:48.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:14:49.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346029 -lt 85899346028 2024-04-25T08:14:49.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:14:49.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182506 2024-04-25T08:14:49.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:14:49.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:14:49.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182506 2024-04-25T08:14:49.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:14:49.530 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182506 2024-04-25T08:14:49.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182506 2024-04-25T08:14:49.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182506' 2024-04-25T08:14:49.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:14:50.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182506 -lt 107374182506 2024-04-25T08:14:50.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:14:50.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:50.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:50.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:14:50.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:50.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:50.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:50.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:50.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:50.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:50.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:51.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:51.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:51.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:51.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:14:51.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:51.790 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:51.791 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:51.792 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:51.793 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:51.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:51.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:51.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:51.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:51.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:51.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:51.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:51.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:51.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:52.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:52.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:52.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:52.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:52.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:14:52.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:52.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:52.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:52.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:52.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:52.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:52.903 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:52.903 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:53.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6552101 2024-04-25T08:14:53.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6552101 '!=' null 2024-04-25T08:14:53.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:53.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:53.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:14:53.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:53.642 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:53.643 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:53.644 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:53.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:53.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:53.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:53.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:53.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:53.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:53.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:53.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:53.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:54.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:54.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:54.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:54.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:54.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:14:54.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:54.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:54.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:54.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:54.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:54.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:54.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:54.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:55.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6552101 2024-04-25T08:14:55.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6552101 '!=' null 2024-04-25T08:14:55.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:55.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:55.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:14:55.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:55.501 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:55.502 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:55.503 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:55.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:55.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:55.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:55.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:55.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:55.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:56.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:56.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:56.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:56.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:56.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:56.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:56.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:57.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5406254 2024-04-25T08:14:57.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5406254 '!=' null 2024-04-25T08:14:57.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:57.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:57.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:14:57.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:57.480 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:57.481 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:57.482 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:57.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:57.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:57.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:57.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:57.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:57.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:57.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:57.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:57.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:58.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:58.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:58.022 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:58.022 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:58.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:58.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:59.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12937328 2024-04-25T08:14:59.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12937328 '!=' null 2024-04-25T08:14:59.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:59.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:59.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:14:59.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:14:59.294 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:59.295 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:14:59.296 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:14:59.297 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:14:59.297 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:14:59.297 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:14:59.297 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:14:59.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:59.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:59.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:59.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:59.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:59.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:59.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:59.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:59.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:59.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:59.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:59.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:00.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:00.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:00.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:00.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:00.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:00.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12230075 2024-04-25T08:15:01.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12230075 '!=' null 2024-04-25T08:15:01.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:01.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:01.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:01.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:01.108 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:01.110 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:01.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:01.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:01.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:01.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:01.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:01.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:01.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:01.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:01.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:01.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:01.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:01.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:01.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:02.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:02.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:02.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20092206 2024-04-25T08:15:02.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20092206 '!=' null 2024-04-25T08:15:02.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:02.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:02.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:02.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:03.029 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:03.030 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:03.031 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:03.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:03.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:03.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:03.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:03.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:03.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:03.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:03.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:03.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:03.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:03.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:03.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:03.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:04.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:04.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:04.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:04.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:04.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:04.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:04.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:04.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:04.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:04.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724248 2024-04-25T08:15:04.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724248 '!=' null 2024-04-25T08:15:04.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:04.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:04.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:04.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:04.830 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:04.830 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:04.830 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:04.830 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:04.831 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:04.832 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:04.833 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:04.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:04.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:04.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:04.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:04.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:04.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:04.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:04.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:04.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:05.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:05.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:05.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:05.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:06.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:06.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:06.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:06.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:06.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:06.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:06.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:06.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:06.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:06.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724283 2024-04-25T08:15:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724283 '!=' null 2024-04-25T08:15:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:06.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:06.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:06.829 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:06.830 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:06.831 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:06.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:06.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:06.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:06.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:06.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:06.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:06.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:06.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:06.946 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:07.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:07.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:07.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:07.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:08.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:08.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:08.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:08.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:08.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:08.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:08.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:08.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:08.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:08.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586645 2024-04-25T08:15:08.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586645 '!=' null 2024-04-25T08:15:08.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:08.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:08.699 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:08.699 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:08.814 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:08.814 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:08.815 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:08.816 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:08.817 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:08.817 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:08.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:08.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:08.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:08.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:08.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:08.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:08.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:08.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:08.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:09.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:09.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:09.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:09.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:10.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:10.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:10.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:11.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724104 2024-04-25T08:15:11.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724104 '!=' null 2024-04-25T08:15:11.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:11.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:11.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:11.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:11.207 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:11.207 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:11.207 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:11.207 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:11.207 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:11.207 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:11.208 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:11.209 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:11.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:11.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:11.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:11.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:11.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:11.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:11.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:11.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:11.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:12.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:13.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586040 2024-04-25T08:15:13.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586040 '!=' null 2024-04-25T08:15:13.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:13.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:13.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:13.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:13.154 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:13.155 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:13.156 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:13.157 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:13.157 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:13.157 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:13.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:13.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:13.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:13.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:13.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:13.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:13.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:13.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:13.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:13.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:13.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:13.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:13.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:14.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:14.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:14.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:14.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:14.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:14.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:14.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:14.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:14.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:15.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723726 2024-04-25T08:15:15.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723726 '!=' null 2024-04-25T08:15:15.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:15.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:15.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:15.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:15.165 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:15.166 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:15.167 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:15.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:15.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:15.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:15.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:15.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:15.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:15.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:15.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:15.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:15.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:15.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:15.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:15.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:16.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:16.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:16.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:16.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723644 2024-04-25T08:15:16.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723644 '!=' null 2024-04-25T08:15:16.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:16.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:16.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:16.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:17.080 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:17.081 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:17.082 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:17.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:17.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:17.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:17.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:17.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:17.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:17.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:17.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:17.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:17.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:17.653 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:17.653 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:17.653 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:18.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:18.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:18.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:19.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585240 2024-04-25T08:15:19.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585240 '!=' null 2024-04-25T08:15:19.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:19.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:19.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:19.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:19.174 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:19.175 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:19.176 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:19.177 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:19.177 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:19.177 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:19.177 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:19.177 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:19.177 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:19.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:19.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:19.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:19.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:19.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:19.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:19.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:19.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:19.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:19.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:19.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:19.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:19.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:20.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:20.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:21.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723111 2024-04-25T08:15:21.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723111 '!=' null 2024-04-25T08:15:21.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:21.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:21.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:21.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:21.216 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:21.216 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:21.216 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:21.216 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:21.217 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:21.219 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:21.219 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:21.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:21.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:21.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:21.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:21.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:21.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:21.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:21.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:21.332 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:21.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:21.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:21.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:21.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:22.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:22.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:22.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585284 2024-04-25T08:15:22.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585284 '!=' null 2024-04-25T08:15:22.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:22.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:22.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:22.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:23.035 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:23.035 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:23.035 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:23.035 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:23.035 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:23.035 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:23.036 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:23.037 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:23.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:23.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:23.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:23.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:23.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:23.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:23.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:23.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:23.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:23.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:23.743 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:23.743 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:23.743 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:24.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:24.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:24.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:24.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723569 2024-04-25T08:15:24.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723569 '!=' null 2024-04-25T08:15:24.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:24.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:24.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:25.075 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:25.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:25.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:25.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:25.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:25.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:25.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:25.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:25.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:25.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:25.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:25.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:25.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:25.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:26.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:26.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:26.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:26.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:26.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:26.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:26.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:26.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:26.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:27.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723651 2024-04-25T08:15:27.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723651 '!=' null 2024-04-25T08:15:27.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:27.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:27.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:27.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:27.213 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:27.213 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:27.213 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:27.213 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:27.213 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:27.214 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:27.215 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:27.216 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:27.216 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:27.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:27.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:27.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:27.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:27.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:27.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:27.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:27.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:27.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:27.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:27.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:27.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:27.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:28.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:28.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:28.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:28.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585619 2024-04-25T08:15:28.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585619 '!=' null 2024-04-25T08:15:28.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:28.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:28.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:28.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:29.104 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:29.104 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:29.104 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:29.105 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:29.107 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:29.107 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:29.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:29.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:29.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:29.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:29.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:29.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:29.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:29.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:29.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:29.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:29.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:29.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:29.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:30.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:30.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:30.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:30.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:30.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:30.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:30.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:30.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:30.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:30.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723273 2024-04-25T08:15:30.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723273 '!=' null 2024-04-25T08:15:30.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:30.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:30.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:30.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:30.949 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:30.950 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:30.951 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:30.952 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:30.952 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:30.952 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:30.952 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:30.952 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:30.952 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:30.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:31.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:31.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:31.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:31.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:31.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:31.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:31.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:31.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:31.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:31.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:31.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:31.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:32.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:32.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:32.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:32.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:32.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:32.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:32.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:32.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:32.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:32.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458868 2024-04-25T08:15:32.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458868 '!=' null 2024-04-25T08:15:32.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:32.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:32.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:32.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:32.765 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:32.766 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:32.767 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:32.768 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:32.768 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:32.768 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:32.768 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:32.768 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:32.768 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:32.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:32.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:32.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:32.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:32.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:32.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:32.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:32.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:32.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:33.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:33.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:33.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:33.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:33.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:33.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:34.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597079 2024-04-25T08:15:34.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597079 '!=' null 2024-04-25T08:15:34.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:34.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:34.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:34.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:34.701 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:34.702 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:34.703 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:34.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:34.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:34.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:34.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:34.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:34.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:34.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:34.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:34.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:35.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:35.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:35.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:35.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:35.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:35.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:35.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:36.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597113 2024-04-25T08:15:36.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597113 '!=' null 2024-04-25T08:15:36.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:36.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:36.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:36.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:36.653 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:36.654 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:36.655 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:36.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:36.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:36.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:36.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:36.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:36.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:36.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:36.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:36.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:37.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:37.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:37.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:37.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:37.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:38.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459026 2024-04-25T08:15:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459026 '!=' null 2024-04-25T08:15:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:38.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:38.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:38.458 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:38.459 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:38.460 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:38.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:38.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:38.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:38.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:38.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:38.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:38.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:38.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:38.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:39.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:39.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:39.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:39.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:39.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:39.688 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:40.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597284 2024-04-25T08:15:40.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597284 '!=' null 2024-04-25T08:15:40.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:40.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:40.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:40.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:40.455 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:40.456 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:40.457 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:40.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:40.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:40.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:40.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:40.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:40.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:40.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:40.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:40.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:40.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:40.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:40.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:40.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:41.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:41.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:42.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459756 2024-04-25T08:15:42.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459756 '!=' null 2024-04-25T08:15:42.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:42.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:42.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:42.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:42.390 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:42.391 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:42.392 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:42.393 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:42.393 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:42.393 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:42.393 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:42.393 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:42.393 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:42.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:42.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:42.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:42.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:42.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:42.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:42.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:42.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:42.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:42.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:42.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:42.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:42.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:43.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724003 2024-04-25T08:15:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724003 '!=' null 2024-04-25T08:15:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:44.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:44.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:44.333 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:44.334 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:44.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:44.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:44.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:44.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:44.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:44.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:44.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:44.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:44.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:44.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:44.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:44.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:44.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:45.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:45.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:46.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723680 2024-04-25T08:15:46.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723680 '!=' null 2024-04-25T08:15:46.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:46.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:46.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:46.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:46.290 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:46.291 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:46.292 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:46.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:46.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:46.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:46.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:46.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:46.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:46.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:46.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:46.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:46.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:46.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:46.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:46.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:47.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:47.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:47.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:47.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:47.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:47.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:47.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:47.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:47.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:47.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459031 2024-04-25T08:15:47.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459031 '!=' null 2024-04-25T08:15:47.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:47.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:47.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:47.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:48.103 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:48.105 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:48.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:48.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:48.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:48.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:48.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:48.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:48.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:48.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:48.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:48.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:48.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:48.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:49.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:49.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:49.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:49.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597095 2024-04-25T08:15:49.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597095 '!=' null 2024-04-25T08:15:49.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:49.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:49.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:49.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:50.020 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:50.021 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:50.022 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:50.023 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:50.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:50.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:50.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:50.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:50.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:50.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:50.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:50.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:51.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:51.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459350 2024-04-25T08:15:51.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459350 '!=' null 2024-04-25T08:15:51.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:51.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:51.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:51.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:51.906 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:51.907 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:51.908 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:51.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:52.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:52.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:52.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:52.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:52.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:52.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:52.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:52.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:52.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:52.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:52.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:52.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:53.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:53.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:53.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:53.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597685 2024-04-25T08:15:53.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597685 '!=' null 2024-04-25T08:15:53.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:53.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:53.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:53.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:53.949 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:53.950 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:53.951 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:53.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:54.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:54.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:54.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:54.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:54.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:54.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:54.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:54.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:54.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:54.494 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:54.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:54.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:55.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:55.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:55.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598029 2024-04-25T08:15:55.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598029 '!=' null 2024-04-25T08:15:55.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:55.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:55.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:55.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:55.980 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:55.980 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:55.981 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:55.982 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:55.983 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:55.983 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:55.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:56.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:56.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:56.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:56.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:56.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:56.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:56.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:56.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:56.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:56.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:56.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:56.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:57.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:57.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:57.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25334745 2024-04-25T08:15:57.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25334745 '!=' null 2024-04-25T08:15:57.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:57.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:57.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:57.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:57.801 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:57.801 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:57.801 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:57.801 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:57.802 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:57.803 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:57.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:57.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:57.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:57.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:57.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:57.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:57.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:57.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:57.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:58.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:15:58.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:58.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:58.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:59.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:59.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598995 2024-04-25T08:15:59.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598995 '!=' null 2024-04-25T08:15:59.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:59.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:59.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:15:59.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:15:59.798 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:15:59.798 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:15:59.798 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:59.798 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:59.799 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:15:59.800 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:15:59.801 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:15:59.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:59.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:59.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:59.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:59.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:59.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:59.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:59.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:59.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:00.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:00.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:00.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:00.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:01.006 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:01.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462233 2024-04-25T08:16:01.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462233 '!=' null 2024-04-25T08:16:01.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:01.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:01.667 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:01.667 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:01.783 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:01.784 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:01.785 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:01.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:01.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:01.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:01.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:01.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:01.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:01.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:01.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:01.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:02.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:02.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:02.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:02.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:03.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:03.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:03.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:03.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599004 2024-04-25T08:16:03.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599004 '!=' null 2024-04-25T08:16:03.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:03.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:03.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:03.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:03.844 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:03.846 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:03.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:03.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:03.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:03.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:03.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:03.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:03.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:03.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:03.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:04.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:04.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:04.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:04.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:05.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:05.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:05.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599141 2024-04-25T08:16:05.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599141 '!=' null 2024-04-25T08:16:05.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:05.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:05.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:05.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:05.970 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:05.970 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:05.970 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:05.970 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:05.970 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:05.970 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:05.971 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:05.972 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:05.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:06.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:06.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:06.086 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:06.086 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:06.086 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:06.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:06.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:06.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:06.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:06.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:06.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:06.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:07.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:07.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:07.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:07.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:07.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:07.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:07.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:07.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:07.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:07.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461822 2024-04-25T08:16:07.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461822 '!=' null 2024-04-25T08:16:07.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:07.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:07.811 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:07.811 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:07.915 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:07.916 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:07.917 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:07.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:08.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:08.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:08.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:08.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:08.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:08.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:08.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:08.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:08.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:08.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:08.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:08.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:09.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:09.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:09.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:09.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725437 2024-04-25T08:16:09.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725437 '!=' null 2024-04-25T08:16:09.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:09.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:09.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:09.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:09.962 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:09.962 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:09.963 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:09.964 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:09.965 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:09.965 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:09.965 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:09.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:10.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:10.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:10.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:10.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:10.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:10.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:10.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:10.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:10.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:10.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:10.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:10.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:11.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:11.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587902 2024-04-25T08:16:11.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587902 '!=' null 2024-04-25T08:16:11.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:11.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:11.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:11.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:11.881 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:11.881 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:11.882 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:11.883 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:11.884 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:11.884 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:11.884 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:11.884 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:11.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:11.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:11.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:11.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:11.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:11.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:11.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:11.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:11.998 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:12.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:12.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:12.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:12.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:13.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:13.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:13.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:13.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725020 2024-04-25T08:16:13.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725020 '!=' null 2024-04-25T08:16:13.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:13.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:13.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:13.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:13.879 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:13.881 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:13.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:13.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:13.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:13.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:13.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:13.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:13.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:13.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:13.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:14.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:14.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:14.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:14.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:15.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:15.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:15.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:15.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:15.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:15.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:15.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:15.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:15.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:15.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724702 2024-04-25T08:16:15.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724702 '!=' null 2024-04-25T08:16:15.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:15.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:15.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:15.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:15.682 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:15.683 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:15.684 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:15.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:15.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:15.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:15.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:15.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:15.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:15.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:15.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:16.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:16.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:16.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:16.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:16.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:16.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:16.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:17.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724702 2024-04-25T08:16:17.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724702 '!=' null 2024-04-25T08:16:17.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:17.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:17.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:17.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:17.554 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:17.554 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:17.555 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:17.556 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:17.557 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:17.557 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:17.557 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:17.557 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:17.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:17.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:17.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:17.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:17.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:17.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:17.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:17.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:17.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:18.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:18.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:18.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:18.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:18.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:18.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:19.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586586 2024-04-25T08:16:19.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586586 '!=' null 2024-04-25T08:16:19.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:19.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:19.436 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:19.436 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:19.551 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:19.551 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:19.551 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:19.551 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:19.551 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:19.551 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:19.552 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:19.553 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:19.554 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:19.554 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:19.554 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:19.554 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:19.554 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:19.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:19.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:19.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:19.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:19.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:19.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:19.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:19.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:19.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:20.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:20.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:20.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:20.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:20.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:21.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724207 2024-04-25T08:16:21.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724207 '!=' null 2024-04-25T08:16:21.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:21.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:21.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:21.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:21.467 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:21.468 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:21.469 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:21.470 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:21.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:21.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:21.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:21.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:21.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:21.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:21.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:21.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:21.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:22.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:22.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:22.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:22.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:22.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:22.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:22.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:22.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:22.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:22.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:22.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:22.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:22.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:23.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586093 2024-04-25T08:16:23.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586093 '!=' null 2024-04-25T08:16:23.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:23.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:23.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:23.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:23.341 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:23.342 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:23.343 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:23.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:23.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:23.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:23.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:23.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:23.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:23.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:23.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:23.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:23.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:23.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:23.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:23.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:24.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:24.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:24.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:24.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:24.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:24.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:25.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723935 2024-04-25T08:16:25.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723935 '!=' null 2024-04-25T08:16:25.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:25.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:25.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:25.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:25.311 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:25.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:25.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:25.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:25.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:25.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:25.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:25.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:25.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:25.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:25.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:25.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:25.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:25.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:26.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:26.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:27.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724233 2024-04-25T08:16:27.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724233 '!=' null 2024-04-25T08:16:27.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:27.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:27.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:27.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:27.354 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:27.355 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:27.356 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:27.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:27.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:27.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:27.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:27.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:27.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:27.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:27.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:27.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:27.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:27.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:27.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:27.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:28.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:28.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:28.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:28.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:29.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460632 2024-04-25T08:16:29.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460632 '!=' null 2024-04-25T08:16:29.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:29.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:29.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:29.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:29.197 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:29.197 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:29.198 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:29.199 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:29.200 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:29.200 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:29.200 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:29.200 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:29.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:29.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:29.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:29.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:29.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:29.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:29.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:29.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:29.314 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:29.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:29.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:29.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:29.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:30.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:30.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:30.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:31.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598133 2024-04-25T08:16:31.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598133 '!=' null 2024-04-25T08:16:31.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:31.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:31.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:31.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:31.148 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:31.148 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:31.148 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:31.148 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:31.149 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:31.150 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:31.151 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:31.151 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:31.151 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:31.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:31.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:31.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:31.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:31.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:31.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:31.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:31.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:31.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:31.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:31.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:31.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:31.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:32.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:32.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:32.432 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:33.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460298 2024-04-25T08:16:33.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460298 '!=' null 2024-04-25T08:16:33.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:33.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:33.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:33.102 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:33.209 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:33.209 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:33.209 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:33.210 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:33.211 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:33.212 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:33.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:33.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:33.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:33.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:33.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:33.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:33.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:33.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:33.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:33.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:33.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:33.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:33.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:34.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:34.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:34.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:34.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:34.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:35.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597816 2024-04-25T08:16:35.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597816 '!=' null 2024-04-25T08:16:35.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:35.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:35.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:35.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:35.126 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "7.0", 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 255, 2024-04-25T08:16:35.127 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:35.128 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:35.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:35.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:35.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:35.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:35.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:35.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:35.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:35.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:35.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:35.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:35.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:35.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:35.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:36.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:36.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:36.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597766 2024-04-25T08:16:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597766 '!=' null 2024-04-25T08:16:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:36.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:36.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:37.016 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:37.017 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:37.018 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:37.018 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:37.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:37.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:37.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:37.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:37.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:37.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:37.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:37.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:37.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:37.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:37.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:37.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:37.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:38.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:38.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:38.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:39.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18868050 2024-04-25T08:16:39.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18868050 '!=' null 2024-04-25T08:16:39.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:39.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:39.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:39.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:39.317 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:39.319 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:39.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:39.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:39.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:39.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:39.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:39.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:39.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:39.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:16:39.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:39.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:39.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:40.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:16:40.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:16:40.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:40.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:40.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:40.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:40.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:40.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:40.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:41.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11792663 2024-04-25T08:16:41.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11792663 '!=' null 2024-04-25T08:16:41.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:41.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:41.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:41.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:41.300 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:41.300 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:41.301 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:41.302 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:41.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:41.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:41.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:41.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:41.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:41.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:41.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:41.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:41.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:41.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:41.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:41.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:41.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:42.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:42.556 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:42.557 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:42.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:42.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:42.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:42.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:42.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:42.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:42.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:42.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:42.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:43.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:43.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:43.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:43.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:43.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:43.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:43.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:44.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22275803 2024-04-25T08:16:44.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22275803 '!=' null 2024-04-25T08:16:44.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:44.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:44.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:44.233 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:44.343 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:44.344 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:44.345 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:44.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:44.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:44.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:44.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:44.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:44.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:44.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:44.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:44.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:44.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:44.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:44.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:44.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:45.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:45.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:46.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10224531 2024-04-25T08:16:46.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10224531 '!=' null 2024-04-25T08:16:46.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:46.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:46.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:46.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:46.454 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:46.455 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:46.456 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:46.456 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:46.456 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:46.456 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:46.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:46.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:46.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:46.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:46.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:46.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:46.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:46.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:46.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:47.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:47.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:47.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:47.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:48.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17821081 2024-04-25T08:16:48.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17821081 '!=' null 2024-04-25T08:16:48.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:48.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:48.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:48.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:48.289 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:48.289 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:48.290 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:48.291 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:48.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:48.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:48.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:48.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:48.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:48.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:48.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:48.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:48.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:48.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:48.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:48.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:48.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:49.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:49.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:49.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:50.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15912556 2024-04-25T08:16:50.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15912556 '!=' null 2024-04-25T08:16:50.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:50.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:50.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:50.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:50.266 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:50.267 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:50.268 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:50.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:50.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:50.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:50.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:50.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:50.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:50.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:50.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:50.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:50.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:50.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:50.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:50.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:51.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:51.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:52.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712494 2024-04-25T08:16:52.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712494 '!=' null 2024-04-25T08:16:52.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:52.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:52.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:52.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:52.243 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:52.243 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:52.243 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:52.243 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:52.244 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:52.245 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:52.246 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:52.246 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:52.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:52.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:52.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:52.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:52.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:52.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:52.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:52.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:52.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:52.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:52.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:52.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:52.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:53.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:53.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:53.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:53.349 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:53.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723838 2024-04-25T08:16:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723838 '!=' null 2024-04-25T08:16:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:54.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:54.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:54.285 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:54.286 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:54.287 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:54.287 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:54.287 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:54.287 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:54.287 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:54.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:54.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:54.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:54.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:54.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:54.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:54.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:54.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:54.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:54.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:54.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:54.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:54.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:55.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:55.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:55.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:55.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:55.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:55.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:55.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:55.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:55.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:56.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723539 2024-04-25T08:16:56.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723539 '!=' null 2024-04-25T08:16:56.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:56.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:56.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:56.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:56.143 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:56.143 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:56.143 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:56.143 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:56.143 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:56.144 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:56.145 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:56.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:56.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:56.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:56.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:56.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:56.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:56.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:56.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:56.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:56.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:56.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:56.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:56.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:57.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:57.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:57.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:57.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:57.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:57.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:57.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:57.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:57.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:58.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459016 2024-04-25T08:16:58.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459016 '!=' null 2024-04-25T08:16:58.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:58.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:58.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:58.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:58.196 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:16:58.196 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:16:58.196 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:58.196 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:58.197 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:16:58.198 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:16:58.199 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:16:58.199 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:16:58.199 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:16:58.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:16:58.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:16:58.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:16:58.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:16:58.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:16:58.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:16:58.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:16:58.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:16:58.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:16:58.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:16:59.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:16:59.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:16:59.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:16:59.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:16:59.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:16:59.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:16:59.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:16:59.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:16:59.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:16:59.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597106 2024-04-25T08:16:59.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597106 '!=' null 2024-04-25T08:16:59.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:16:59.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:16:59.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:16:59.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:00.053 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:00.054 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:00.055 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:00.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:00.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:00.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:00.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:00.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:00.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:00.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:00.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:00.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:00.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:00.625 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:00.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:00.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:01.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:01.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:01.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:01.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:01.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:01.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:01.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:01.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:01.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459295 2024-04-25T08:17:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459295 '!=' null 2024-04-25T08:17:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:01.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:01.973 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:02.082 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:02.083 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:02.084 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:02.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:02.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:02.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:02.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:02.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:02.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:02.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:02.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:02.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:02.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:02.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:02.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:02.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:03.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:03.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:03.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:03.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597593 2024-04-25T08:17:03.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597593 '!=' null 2024-04-25T08:17:03.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:03.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:03.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:03.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:03.870 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:03.870 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:03.870 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:03.870 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:03.870 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:03.870 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:03.872 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:03.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:03.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:03.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:03.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:03.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:03.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:03.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:03.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:03.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:04.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:04.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:04.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:04.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:05.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:05.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:05.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:05.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:05.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:05.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:05.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:05.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:05.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:05.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597908 2024-04-25T08:17:05.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597908 '!=' null 2024-04-25T08:17:05.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:05.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:05.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:05.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:05.951 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:05.952 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:05.953 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:05.954 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:05.955 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:05.955 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:05.955 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:05.955 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:05.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:06.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:06.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:06.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:06.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:06.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:06.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:06.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:06.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:06.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:06.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:06.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:06.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:07.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:07.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:07.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:07.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:07.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460821 2024-04-25T08:17:07.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460821 '!=' null 2024-04-25T08:17:07.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:07.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:07.663 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:07.663 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:07.773 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:07.774 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:07.775 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:07.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:07.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:07.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:07.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:07.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:07.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:07.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:07.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:07.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:08.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:08.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:08.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:08.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:09.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:09.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:09.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:09.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:09.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:09.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724759 2024-04-25T08:17:09.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724759 '!=' null 2024-04-25T08:17:09.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:09.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:09.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:09.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:09.729 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:09.730 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:09.731 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:09.732 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:09.732 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:09.732 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:09.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:09.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:09.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:09.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:09.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:09.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:09.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:09.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:09.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:10.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:10.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:10.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:10.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:10.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:10.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:10.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:10.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:10.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:10.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:10.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:10.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:10.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:11.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587392 2024-04-25T08:17:11.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587392 '!=' null 2024-04-25T08:17:11.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:11.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:11.516 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:11.516 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:11.625 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:11.626 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:11.627 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:11.627 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:11.627 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:11.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:11.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:11.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:11.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:11.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:11.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:11.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:11.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:11.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:12.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:12.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:12.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:12.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:12.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:12.791 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:13.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587392 2024-04-25T08:17:13.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587392 '!=' null 2024-04-25T08:17:13.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:13.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:13.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:13.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:13.456 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:13.457 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:13.458 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:13.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:13.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:13.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:13.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:13.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:13.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:13.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:13.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:13.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:13.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:13.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:13.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:13.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:14.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:14.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:14.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:14.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:14.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:14.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:14.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:14.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:14.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:15.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724917 2024-04-25T08:17:15.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724917 '!=' null 2024-04-25T08:17:15.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:15.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:15.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:15.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:15.374 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:15.375 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:15.376 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:15.376 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:15.376 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:15.376 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:15.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:15.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:15.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:15.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:15.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:15.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:15.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:15.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:15.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:15.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:15.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:16.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:16.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:16.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:17.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724819 2024-04-25T08:17:17.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724819 '!=' null 2024-04-25T08:17:17.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:17.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:17.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:17.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:17.206 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:17.206 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:17.206 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:17.206 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:17.207 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:17.208 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:17.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:17.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:17.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:17.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:17.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:17.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:17.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:17.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:17.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:17.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:17.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:17.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:17.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:18.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:18.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:18.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:19.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713506 2024-04-25T08:17:19.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713506 '!=' null 2024-04-25T08:17:19.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:19.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:19.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:19.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:19.167 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:19.167 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:19.168 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:19.169 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:19.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:19.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:19.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:19.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:19.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:19.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:19.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:19.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:19.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:19.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:19.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:19.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:19.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850700 2024-04-25T08:17:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850700 '!=' null 2024-04-25T08:17:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:21.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:21.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:21.262 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:21.263 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:21.264 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:21.264 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:21.264 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:21.264 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:21.264 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:21.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:21.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:21.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:21.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:21.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:21.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:21.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:21.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:21.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:21.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:21.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:21.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:21.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:23.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585702 2024-04-25T08:17:23.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585702 '!=' null 2024-04-25T08:17:23.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:23.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:23.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:23.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:23.411 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:23.412 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:23.413 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:23.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:23.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:23.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:23.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:23.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:23.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:23.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:23.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:23.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:23.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:23.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:23.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:23.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:24.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:24.630 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723516 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723516 '!=' null 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:25.385 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:25.385 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:25.386 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:25.387 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:25.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:25.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:25.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:25.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:25.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:25.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:25.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:25.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:25.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:25.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:25.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:25.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:25.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:26.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:26.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:27.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723636 2024-04-25T08:17:27.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723636 '!=' null 2024-04-25T08:17:27.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:27.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:27.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:27.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:27.271 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:27.271 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:27.272 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:27.273 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:27.274 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:27.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:27.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:27.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:27.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:27.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:27.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:27.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:27.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:27.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:27.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:27.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:27.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:27.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:28.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:28.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:28.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:28.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:28.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:28.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:28.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:28.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:28.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:29.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585126 2024-04-25T08:17:29.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585126 '!=' null 2024-04-25T08:17:29.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:29.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:29.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:29.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:29.269 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:29.270 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:29.271 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:29.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:29.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:29.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:29.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:29.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:29.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:29.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:29.834 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:30.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:30.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:30.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597070 2024-04-25T08:17:30.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597070 '!=' null 2024-04-25T08:17:30.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:30.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:30.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:30.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:31.106 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:31.107 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:31.109 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:31.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:31.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:31.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:31.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:31.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:31.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:31.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:31.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:31.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:31.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:31.649 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:31.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:31.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:32.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:32.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:32.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459067 2024-04-25T08:17:32.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459067 '!=' null 2024-04-25T08:17:32.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:32.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:32.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:32.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:32.896 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:32.896 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:32.897 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:32.898 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:32.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:33.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:33.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:33.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:33.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:33.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:33.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:33.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:33.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:33.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:33.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:33.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:33.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:34.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:34.102 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:34.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:34.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724009 2024-04-25T08:17:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724009 '!=' null 2024-04-25T08:17:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:34.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:34.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:34.974 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:34.975 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:34.976 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:34.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:35.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:35.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:35.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:35.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:35.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:35.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:35.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:35.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:35.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:35.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:35.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:35.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:36.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:36.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:36.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:36.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:36.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:36.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:36.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:36.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:36.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724190 2024-04-25T08:17:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724190 '!=' null 2024-04-25T08:17:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:36.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:36.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:36.862 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:36.863 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:36.864 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:36.864 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:36.864 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:36.864 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:36.864 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:36.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:36.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:36.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:36.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:36.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:36.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:36.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:36.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:36.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:37.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:37.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:37.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:37.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:38.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:38.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586567 2024-04-25T08:17:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586567 '!=' null 2024-04-25T08:17:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:38.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:38.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:38.966 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:38.966 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:38.966 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:38.966 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:38.967 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:38.968 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:38.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:39.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:39.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:39.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:39.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:39.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:39.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:39.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:39.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:39.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:39.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:39.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:39.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:40.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:40.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:40.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724531 2024-04-25T08:17:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724531 '!=' null 2024-04-25T08:17:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:40.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:40.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:41.003 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:41.003 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:41.003 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:41.003 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:41.003 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:41.003 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:41.004 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:41.005 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:41.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:41.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:41.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:41.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:41.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:41.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:41.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:41.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:41.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:41.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:41.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:41.634 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:41.634 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:42.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:42.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:42.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586914 2024-04-25T08:17:42.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586914 '!=' null 2024-04-25T08:17:42.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:42.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:42.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:42.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:43.059 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:43.060 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:43.061 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:43.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:43.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:43.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:43.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:43.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:43.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:43.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:43.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:43.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:43.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:43.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:43.731 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:43.731 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:44.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:44.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:45.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724588 2024-04-25T08:17:45.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724588 '!=' null 2024-04-25T08:17:45.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:45.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:45.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:45.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:45.113 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:45.115 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:45.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:45.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:45.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:45.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:45.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:45.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:45.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:45.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:45.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:45.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:45.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:45.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:45.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:46.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:46.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:46.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:47.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724673 2024-04-25T08:17:47.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724673 '!=' null 2024-04-25T08:17:47.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:47.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:47.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:47.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:47.253 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:47.254 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:47.255 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:47.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:47.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:47.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:47.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:47.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:47.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:47.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:47.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:47.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:47.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:47.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:47.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:47.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:48.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:48.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:49.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586846 2024-04-25T08:17:49.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586846 '!=' null 2024-04-25T08:17:49.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:49.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:49.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:49.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:49.371 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:49.372 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:49.373 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:49.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:49.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:49.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:49.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:49.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:49.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:49.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:49.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:49.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:50.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:50.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:50.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:50.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:50.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:50.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:50.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:51.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724261 2024-04-25T08:17:51.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724261 '!=' null 2024-04-25T08:17:51.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:51.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:51.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:51.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:51.368 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:51.369 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:51.370 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:51.370 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:51.370 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:51.370 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:51.370 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:51.370 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:51.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:51.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:51.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:51.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:51.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:51.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:51.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:51.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:51.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:52.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:52.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:52.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:52.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:52.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:52.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:52.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:52.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:52.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:52.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:52.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:52.585 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:52.585 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:53.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712643 2024-04-25T08:17:53.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712643 '!=' null 2024-04-25T08:17:53.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:53.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:53.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:53.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:53.322 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:53.324 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:53.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:53.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:53.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:53.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:53.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:53.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:53.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:53.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:53.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:53.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:53.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:53.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:53.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:54.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:54.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:54.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:54.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:54.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:54.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:55.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850410 2024-04-25T08:17:55.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850410 '!=' null 2024-04-25T08:17:55.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:55.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:55.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:55.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:55.164 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:55.164 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:55.164 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:55.164 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:55.164 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:55.165 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:55.166 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:55.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:55.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:55.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:55.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:55.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:55.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:55.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:55.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:55.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:55.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:55.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:55.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:55.708 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:56.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:56.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850209 2024-04-25T08:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850209 '!=' null 2024-04-25T08:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:57.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:57.117 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:57.117 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:57.118 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:57.119 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:57.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:57.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:57.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:57.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:57.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:57.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:57.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:57.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:57.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:57.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:57.661 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:57.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:57.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:17:58.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:17:58.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22711911 2024-04-25T08:17:58.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22711911 '!=' null 2024-04-25T08:17:58.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:17:58.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:58.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:17:58.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:58.923 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:17:58.924 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:17:58.925 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:17:58.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:17:59.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:17:59.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:59.037 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:59.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:59.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:59.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:59.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:59.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:59.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:59.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:59.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:59.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:00.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:00.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:00.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:00.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:00.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:00.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:00.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:00.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:00.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:00.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850282 2024-04-25T08:18:00.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850282 '!=' null 2024-04-25T08:18:00.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:00.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:00.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:00.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:00.902 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:00.902 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:00.902 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:00.903 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:00.904 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:00.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:01.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:01.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:01.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:01.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:01.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:01.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:01.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:01.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:01.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:01.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:01.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:01.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:02.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:02.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712291 2024-04-25T08:18:02.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712291 '!=' null 2024-04-25T08:18:02.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:02.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:02.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:02.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:03.005 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:03.006 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:03.007 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:03.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:03.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:03.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:03.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:03.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:03.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:03.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:03.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:03.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:03.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:03.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:03.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:03.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:04.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:04.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:04.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724274 2024-04-25T08:18:04.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724274 '!=' null 2024-04-25T08:18:04.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:04.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:04.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:04.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:04.979 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:04.980 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:04.981 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:04.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:05.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:05.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:05.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:05.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:05.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:05.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:05.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:05.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:05.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:05.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:05.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:05.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:06.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:06.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:06.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724448 2024-04-25T08:18:06.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724448 '!=' null 2024-04-25T08:18:06.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:06.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:06.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:06.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:06.972 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:06.973 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:06.974 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:06.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:07.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:07.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:07.086 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:07.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:07.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:07.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:07.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:07.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:07.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:07.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:07.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:07.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:08.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:08.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:09.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587491 2024-04-25T08:18:09.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587491 '!=' null 2024-04-25T08:18:09.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:09.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:09.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:09.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:09.209 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:09.210 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:09.211 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:09.211 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:09.211 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:09.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:09.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:09.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:09.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:09.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:09.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:09.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:09.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:09.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:09.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:09.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:09.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:09.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:10.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:10.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:10.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:11.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725004 2024-04-25T08:18:11.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725004 '!=' null 2024-04-25T08:18:11.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:11.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:11.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:11.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:11.153 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:11.154 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:11.155 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:11.155 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:11.155 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:11.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:11.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:11.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:11.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:11.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:11.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:11.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:11.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:11.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:11.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:11.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:11.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:11.736 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:12.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:12.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:12.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:12.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:12.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:12.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:12.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:12.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:12.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:12.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713645 2024-04-25T08:18:12.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713645 '!=' null 2024-04-25T08:18:12.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:12.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:12.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:12.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:13.046 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:13.046 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:13.047 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:13.048 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:13.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:13.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:13.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:13.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:13.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:13.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:13.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:13.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:13.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:13.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:13.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:13.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:13.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:14.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:14.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:14.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:14.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:14.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:14.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:14.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:14.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:14.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:14.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851115 2024-04-25T08:18:14.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851115 '!=' null 2024-04-25T08:18:14.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:14.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:14.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:14.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:15.095 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:15.097 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:15.100 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:15.100 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:15.100 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:15.100 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:15.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:15.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:15.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:15.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:15.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:15.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:15.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:15.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:15.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:15.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:15.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:15.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:15.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:16.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:16.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:16.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:16.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:16.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:16.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:16.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:16.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:16.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:17.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850879 2024-04-25T08:18:17.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850879 '!=' null 2024-04-25T08:18:17.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:17.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:17.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:17.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:17.131 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:17.132 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:17.133 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:17.133 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:17.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:17.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:17.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:17.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:17.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:17.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:17.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:17.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:17.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:17.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:17.818 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:17.822 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:17.822 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:18.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:18.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:18.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:19.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712723 2024-04-25T08:18:19.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712723 '!=' null 2024-04-25T08:18:19.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:19.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:19.074 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:19.074 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:19.183 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:19.183 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:19.183 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:19.183 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:19.184 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:19.185 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:19.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:19.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:19.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:19.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:19.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:19.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:19.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:19.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:19.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:19.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:19.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:19.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:20.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:20.267 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:20.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850268 2024-04-25T08:18:20.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850268 '!=' null 2024-04-25T08:18:20.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:20.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:20.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:20.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:21.005 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:21.005 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:21.005 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:21.005 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:21.006 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:21.007 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:21.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:21.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:21.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:21.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:21.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:21.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:21.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:21.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:21.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:21.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:21.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:21.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:21.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:22.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:22.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:22.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838709 2024-04-25T08:18:22.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838709 '!=' null 2024-04-25T08:18:22.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:22.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:22.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:22.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:23.030 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:23.030 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:23.030 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:23.031 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:23.032 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:23.033 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:23.033 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:23.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:23.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:23.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:23.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:23.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:23.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:23.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:23.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:23.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:23.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:23.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:23.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:23.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:24.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:24.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:24.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:24.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:24.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:24.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:24.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:24.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:24.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:24.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850659 2024-04-25T08:18:24.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850659 '!=' null 2024-04-25T08:18:24.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:24.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:24.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:24.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:25.036 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:25.037 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:25.038 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:25.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:25.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:25.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:25.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:25.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:25.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:25.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:25.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:25.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:25.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:25.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:25.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:25.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:26.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:26.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:26.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851031 2024-04-25T08:18:26.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851031 '!=' null 2024-04-25T08:18:26.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:26.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:26.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:26.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:26.878 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:26.879 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:26.880 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:26.880 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:26.880 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:26.880 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:26.880 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:26.880 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:26.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:26.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:26.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:26.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:26.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:27.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:27.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:27.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:27.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:27.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:27.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:27.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:28.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713863 2024-04-25T08:18:28.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713863 '!=' null 2024-04-25T08:18:28.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:28.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:28.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:28.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [ 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 190, 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:28.677 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "1.0", 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:28.678 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:28.679 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:28.679 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:28.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:28.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:28.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:28.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:28.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:28.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:28.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:28.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:28.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:29.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:29.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:29.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:29.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:30.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:30.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:30.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:30.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:30.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:30.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:30.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:30.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:30.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:30.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851499 2024-04-25T08:18:30.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851499 '!=' null 2024-04-25T08:18:30.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:30.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:30.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:30.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:31.095 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:31.095 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:31.095 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:31.095 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:31.096 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:31.097 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:31.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:31.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:31.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:31.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:31.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:31.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:31.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:31.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:31.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:31.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:31.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:31.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:31.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:32.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:32.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:32.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:33.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588137 2024-04-25T08:18:33.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588137 '!=' null 2024-04-25T08:18:33.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:33.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:33.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:33.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:33.265 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:33.265 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:33.265 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:33.265 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:33.265 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:33.266 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:33.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:33.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:33.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:33.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:33.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:33.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:33.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:33.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:33.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:33.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:33.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:33.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:33.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:34.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:34.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:34.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:35.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587963 2024-04-25T08:18:35.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587963 '!=' null 2024-04-25T08:18:35.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:35.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:35.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:35.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:35.306 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:35.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:35.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:35.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:35.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:35.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:35.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:35.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:35.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:35.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:35.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:35.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:35.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:35.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:36.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:36.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793775 2024-04-25T08:18:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793775 '!=' null 2024-04-25T08:18:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:37.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:37.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:37.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:37.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:37.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:37.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:37.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:37.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:37.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:37.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:37.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:37.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:37.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:37.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:38.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:38.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-04-25T08:18:38.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:18:38.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-25T08:18:38.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:38.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:38.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:38.554 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:38.554 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:38.554 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:38.554 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:38.554 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:38.554 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:38.555 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:38.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:38.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:38.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:38.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:38.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:38.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:38.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:38.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:38.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:39.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:39.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:39.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:39.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:39.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:39.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:39.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:40.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20054690 2024-04-25T08:18:40.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20054690 '!=' null 2024-04-25T08:18:40.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:40.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:40.287 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:40.287 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:40.396 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:40.397 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:40.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:40.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:40.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:40.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:40.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:40.511 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:40.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:40.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:41.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:41.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:41.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:41.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:41.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:41.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:42.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27934806 2024-04-25T08:18:42.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27934806 '!=' null 2024-04-25T08:18:42.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:42.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:42.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:42.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:42.301 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:42.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:42.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:42.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:42.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:42.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:42.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:42.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:42.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:42.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:42.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:42.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:42.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:42.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:43.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:43.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:44.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471979 2024-04-25T08:18:44.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471979 '!=' null 2024-04-25T08:18:44.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:44.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:44.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:44.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:44.249 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:44.250 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:44.250 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:44.250 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:44.250 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:44.250 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:44.250 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:44.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:44.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:44.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:44.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:44.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:44.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:44.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:44.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:44.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:44.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:44.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:44.814 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:44.814 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:45.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:45.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:46.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471936 2024-04-25T08:18:46.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471936 '!=' null 2024-04-25T08:18:46.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:46.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:46.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:46.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:46.217 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:46.218 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:46.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:46.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:46.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:46.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:46.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:46.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:46.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:46.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:46.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:46.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:46.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:46.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:46.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:47.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25334052 2024-04-25T08:18:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25334052 '!=' null 2024-04-25T08:18:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:48.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:48.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:48.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:48.244 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:48.245 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:48.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:48.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:48.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:48.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:48.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:48.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:48.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:48.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:48.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:48.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:48.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:48.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:48.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:49.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:49.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:49.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598259 2024-04-25T08:18:49.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598259 '!=' null 2024-04-25T08:18:49.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:49.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:49.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:49.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:50.094 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:50.094 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:50.095 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:50.096 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:50.096 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:50.096 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:50.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:50.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:50.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:50.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:50.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:50.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:50.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:50.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:50.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:50.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:50.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:50.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:50.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:51.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:51.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:51.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:52.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460296 2024-04-25T08:18:52.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460296 '!=' null 2024-04-25T08:18:52.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:52.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:52.097 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:52.097 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:52.207 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:52.208 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:52.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:52.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:52.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:52.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:52.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:52.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:52.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:52.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:52.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:52.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:52.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:52.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:52.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:53.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:53.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:53.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:54.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724062 2024-04-25T08:18:54.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724062 '!=' null 2024-04-25T08:18:54.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:54.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:54.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:54.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:54.300 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:54.302 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:54.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:54.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:54.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:54.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:54.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:54.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:54.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:54.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:54.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:54.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:54.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:54.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:54.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:55.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:55.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:56.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724025 2024-04-25T08:18:56.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724025 '!=' null 2024-04-25T08:18:56.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:56.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:56.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:56.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:56.327 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:56.327 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:56.327 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:56.328 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:56.329 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:56.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:56.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:56.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:56.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:56.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:56.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:56.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:56.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:56.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:56.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:56.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:56.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:56.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:57.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:57.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:57.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:57.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:57.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:57.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:57.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:57.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:57.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22711867 2024-04-25T08:18:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22711867 '!=' null 2024-04-25T08:18:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:58.100 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:18:58.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:58.215 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:18:58.215 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:18:58.215 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:18:58.216 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:18:58.217 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:18:58.217 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:18:58.217 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:18:58.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:58.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:58.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:58.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:58.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:58.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:58.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:58.332 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:58.332 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:58.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:58.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:59.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:59.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850258 2024-04-25T08:19:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850258 '!=' null 2024-04-25T08:19:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:00.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:00.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:00.180 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:00.181 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:00.182 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:00.182 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:00.182 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:00.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:00.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:00.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:00.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:00.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:00.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:00.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:00.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:00.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:00.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:00.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:00.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:00.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:01.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:01.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:01.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712096 2024-04-25T08:19:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712096 '!=' null 2024-04-25T08:19:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:02.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:02.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:02.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:02.159 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:02.159 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:02.159 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:02.159 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:02.159 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:02.160 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:02.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:02.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:02.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:02.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:02.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:02.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:02.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:02.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:02.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:02.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:02.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:02.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:02.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:03.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:03.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:03.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:03.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850245 2024-04-25T08:19:03.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850245 '!=' null 2024-04-25T08:19:03.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:03.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:03.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:03.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:04.038 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:04.039 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:04.040 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:04.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:04.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:04.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:04.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:04.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:04.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:04.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:04.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:04.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:04.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:04.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:04.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:04.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:05.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:05.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:05.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:05.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850581 2024-04-25T08:19:05.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850581 '!=' null 2024-04-25T08:19:05.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:05.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:05.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:05.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:05.818 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:05.818 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:05.818 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:05.819 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:05.820 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:05.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:05.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:05.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:05.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:05.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:05.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:05.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:05.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:05.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:06.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:06.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:06.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:06.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:07.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:07.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:07.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:07.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586503 2024-04-25T08:19:07.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586503 '!=' null 2024-04-25T08:19:07.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:07.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:07.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:07.701 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:07.814 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:07.815 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:07.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:07.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:07.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:07.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:07.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:07.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:07.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:07.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:07.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:08.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:08.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:08.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:08.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:08.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:08.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:08.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:09.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586503 2024-04-25T08:19:09.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586503 '!=' null 2024-04-25T08:19:09.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:09.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:09.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:09.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:09.657 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:09.658 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:09.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:09.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:09.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:09.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:09.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:09.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:09.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:09.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:09.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:10.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:10.206 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:10.206 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:10.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:10.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:10.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:10.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:11.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586283 2024-04-25T08:19:11.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586283 '!=' null 2024-04-25T08:19:11.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:11.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:11.612 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:11.612 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:11.723 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:11.724 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:11.725 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:11.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:11.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:11.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:11.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:11.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:11.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:11.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:11.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:11.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:12.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:12.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:12.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:12.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:12.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:12.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:12.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:13.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586283 2024-04-25T08:19:13.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586283 '!=' null 2024-04-25T08:19:13.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:13.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:13.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:13.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:13.591 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:13.591 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:13.591 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:13.592 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:13.593 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:13.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:13.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:13.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:13.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:13.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:13.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:13.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:13.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:13.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:14.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:14.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:14.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:14.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:14.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:14.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:14.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:14.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:14.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:14.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:14.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:14.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:14.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724164 2024-04-25T08:19:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724164 '!=' null 2024-04-25T08:19:15.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:15.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:15.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:15.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:15.458 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:15.458 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:15.459 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:15.460 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:15.460 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:15.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:15.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:15.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:15.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:15.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:15.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:15.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:15.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:15.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:16.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:16.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:16.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:16.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:16.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:17.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724135 2024-04-25T08:19:17.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724135 '!=' null 2024-04-25T08:19:17.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:17.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:17.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:17.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:17.392 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:17.392 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:17.392 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:17.392 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:17.393 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:17.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:17.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:17.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:17.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:17.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:17.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:17.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:17.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:17.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:18.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:18.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:18.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:18.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:18.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:18.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:18.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:19.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586006 2024-04-25T08:19:19.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586006 '!=' null 2024-04-25T08:19:19.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:19.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:19.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:19.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:19.310 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:19.311 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:19.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:19.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:19.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:19.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:19.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:19.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:19.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:19.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:19.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:19.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:19.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:19.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:19.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:20.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:20.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:20.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:21.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850210 2024-04-25T08:19:21.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850210 '!=' null 2024-04-25T08:19:21.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:21.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:21.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:21.195 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:21.306 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:21.308 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:21.308 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:21.308 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:21.308 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:21.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:21.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:21.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:21.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:21.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:21.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:21.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:21.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:21.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:21.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:21.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:21.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:21.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:22.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:22.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:22.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:22.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:22.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:22.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:22.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:22.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:22.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:23.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712226 2024-04-25T08:19:23.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712226 '!=' null 2024-04-25T08:19:23.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:23.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:23.084 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:23.084 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:23.195 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:23.195 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:23.195 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:23.195 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:23.195 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:23.196 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:23.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:23.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:23.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:23.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:23.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:23.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:23.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:23.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:23.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:23.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:23.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:23.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:23.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:24.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:24.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:24.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:24.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:24.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:24.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:24.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:24.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:24.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:25.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724307 2024-04-25T08:19:25.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724307 '!=' null 2024-04-25T08:19:25.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:25.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:25.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:25.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:25.134 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:25.135 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:25.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:25.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:25.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:25.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:25.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:25.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:25.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:25.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:25.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:25.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:25.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:25.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:25.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:26.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:26.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:26.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:26.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:26.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:26.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724334 2024-04-25T08:19:26.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724334 '!=' null 2024-04-25T08:19:26.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:26.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:26.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:26.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:26.965 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:26.966 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:26.967 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:26.967 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:26.967 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:26.967 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:26.967 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:26.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:27.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:27.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:27.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:27.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:27.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:27.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:27.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:27.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:27.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:27.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:27.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:27.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:28.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:28.084 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:28.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713231 2024-04-25T08:19:28.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713231 '!=' null 2024-04-25T08:19:28.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:28.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:28.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:28.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:28.799 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:28.800 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:28.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:28.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:28.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:28.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:28.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:28.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:28.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:28.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:28.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:29.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:29.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:29.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:29.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:30.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:30.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:30.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:30.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:30.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:30.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:30.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:30.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:30.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:30.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851005 2024-04-25T08:19:30.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851005 '!=' null 2024-04-25T08:19:30.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:30.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:30.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:30.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:30.778 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:30.779 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:30.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:30.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:30.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:30.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:30.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:30.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:30.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:30.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:30.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:31.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:31.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:31.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:31.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:32.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:32.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:32.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713160 2024-04-25T08:19:32.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713160 '!=' null 2024-04-25T08:19:32.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:32.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:32.866 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:32.866 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:32.980 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:32.981 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:32.981 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:32.981 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:32.981 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:32.981 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:32.981 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:32.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:33.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:33.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:33.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:33.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:33.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:33.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:33.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:33.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:33.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:33.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:33.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:33.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:34.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:34.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:34.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:34.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850706 2024-04-25T08:19:34.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850706 '!=' null 2024-04-25T08:19:34.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:34.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:34.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:34.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:34.992 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:34.993 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:35.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:35.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:35.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:35.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:35.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:35.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:35.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:35.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:35.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:35.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:35.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:35.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:35.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:36.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:36.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850565 2024-04-25T08:19:36.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850565 '!=' null 2024-04-25T08:19:36.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:36.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:36.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:36.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:36.960 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:36.961 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:36.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:37.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:37.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:37.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:37.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:37.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:37.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:37.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:37.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:37.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:37.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:37.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:37.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:38.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:38.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:38.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:38.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:38.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:38.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:38.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:38.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:38.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:39.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712695 2024-04-25T08:19:39.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712695 '!=' null 2024-04-25T08:19:39.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:39.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:39.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:39.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:39.132 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:39.133 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:39.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:39.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:39.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:39.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:39.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:39.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:39.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:39.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:39.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:39.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:39.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:39.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:39.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:40.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:40.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:40.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:40.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:40.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:40.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:40.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:40.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:40.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:40.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724079 2024-04-25T08:19:40.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724079 '!=' null 2024-04-25T08:19:40.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:40.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:40.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:40.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:41.103 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:41.104 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:41.104 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:41.104 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:41.104 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:41.104 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:41.104 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:41.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:41.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:41.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:41.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:41.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:41.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:41.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:41.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:41.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:41.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:41.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:41.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:41.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:42.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:42.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:42.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:42.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:42.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:42.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:42.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:42.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:42.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:42.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585909 2024-04-25T08:19:42.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585909 '!=' null 2024-04-25T08:19:42.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:42.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:42.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:42.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:43.033 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:43.034 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:43.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:43.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:43.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:43.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:43.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:43.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:43.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:43.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:43.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:43.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:43.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:43.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:43.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:44.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:44.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:44.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:44.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723984 2024-04-25T08:19:44.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723984 '!=' null 2024-04-25T08:19:44.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:44.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:44.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:44.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:45.027 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:45.028 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:45.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:45.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:45.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:45.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:45.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:45.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:45.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:45.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:45.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:45.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:45.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:45.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:45.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:46.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:46.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723926 2024-04-25T08:19:46.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723926 '!=' null 2024-04-25T08:19:46.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:46.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:46.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:46.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:47.031 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:47.032 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:47.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:47.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:47.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:47.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:47.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:47.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:47.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:47.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:47.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:47.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:47.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:47.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:47.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:48.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:48.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:48.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:49.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586400 2024-04-25T08:19:49.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586400 '!=' null 2024-04-25T08:19:49.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:49.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:49.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:49.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:49.177 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:49.178 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:49.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:49.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:49.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:49.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:49.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:49.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:49.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:49.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:49.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:49.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:49.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:50.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:50.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:50.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724499 2024-04-25T08:19:50.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724499 '!=' null 2024-04-25T08:19:50.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:50.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:50.950 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:50.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:51.060 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:51.061 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:51.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:51.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:51.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:51.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:51.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:51.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:51.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:51.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:51.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:51.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:51.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:51.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:51.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:52.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:52.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:52.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:52.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:52.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:52.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586861 2024-04-25T08:19:52.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586861 '!=' null 2024-04-25T08:19:52.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:52.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:52.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:52.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:53.044 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:53.045 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:53.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:53.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:53.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:53.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:53.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:53.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:53.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:53.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:53.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:53.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:53.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:53.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:53.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:54.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:54.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:54.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:54.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:54.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:54.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:54.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:54.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:54.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:54.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724526 2024-04-25T08:19:54.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724526 '!=' null 2024-04-25T08:19:54.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:54.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:54.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:54.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:55.046 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:55.047 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:55.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:55.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:55.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:55.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:55.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:55.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:55.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:55.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:55.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:55.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:55.589 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:55.589 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:55.589 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:56.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:56.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:56.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:56.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724509 2024-04-25T08:19:56.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724509 '!=' null 2024-04-25T08:19:56.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:56.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:56.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:56.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:56.952 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:56.952 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:56.952 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:56.952 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:56.953 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:56.954 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:56.954 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:56.954 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:56.954 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:56.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:57.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:57.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:57.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:57.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:57.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:57.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:57.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:57.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:57.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:57.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:57.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:57.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:58.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:19:58.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:58.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:58.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713659 2024-04-25T08:19:58.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713659 '!=' null 2024-04-25T08:19:58.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:58.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:58.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:58.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:19:58.917 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:19:58.918 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:19:58.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:59.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:59.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:59.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:59.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:59.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:59.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:59.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:59.034 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:59.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:19:59.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:59.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:59.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:00.313 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850930 2024-04-25T08:20:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850930 '!=' null 2024-04-25T08:20:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:00.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:00.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:00.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:01.022 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:01.023 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:01.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:01.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:01.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:01.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:01.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:01.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:01.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:01.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:01.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:01.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:01.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:01.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:01.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:02.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:02.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:02.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:02.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:02.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:02.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713239 2024-04-25T08:20:02.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713239 '!=' null 2024-04-25T08:20:02.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:02.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:02.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:02.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:03.026 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:03.026 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:03.026 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:03.026 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:03.026 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:03.026 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:03.027 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:03.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:03.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:03.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:03.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:03.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:03.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:03.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:03.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:03.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:03.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:03.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:03.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:04.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:04.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:04.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850800 2024-04-25T08:20:04.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850800 '!=' null 2024-04-25T08:20:04.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:04.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:04.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:04.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:05.074 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:05.075 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:05.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:05.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:05.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:05.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:05.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:05.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:05.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:05.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:05.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:05.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:05.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:05.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:05.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:06.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:06.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:06.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:06.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:06.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:06.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:06.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:06.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:06.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851069 2024-04-25T08:20:06.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851069 '!=' null 2024-04-25T08:20:06.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:06.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:06.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:06.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:07.087 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:07.088 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:07.089 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:07.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:07.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:07.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:07.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:07.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:07.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:07.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:07.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:07.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:07.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:07.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:07.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:07.626 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:08.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:08.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:08.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:08.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713363 2024-04-25T08:20:08.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713363 '!=' null 2024-04-25T08:20:08.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:08.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:08.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:08.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:08.943 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:08.944 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:08.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:09.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:09.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:09.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:09.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:09.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:09.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:09.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:09.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:09.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:09.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:09.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:09.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:10.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:10.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:10.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:10.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724598 2024-04-25T08:20:10.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724598 '!=' null 2024-04-25T08:20:10.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:10.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:10.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:10.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:10.909 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:10.909 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:10.909 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:10.909 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:10.909 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:10.910 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:10.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:11.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:11.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:11.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:11.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:11.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:11.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:11.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:11.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:11.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:11.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:11.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:11.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:12.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:12.206 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:12.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:12.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586832 2024-04-25T08:20:12.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586832 '!=' null 2024-04-25T08:20:12.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:12.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:12.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:12.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:12.864 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:12.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:12.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:12.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:12.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:12.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:12.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:12.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:12.980 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:12.980 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:13.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:13.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:13.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:13.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:14.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:14.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:14.022 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:14.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724104 2024-04-25T08:20:14.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724104 '!=' null 2024-04-25T08:20:14.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:14.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:14.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:14.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:14.735 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:14.736 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:14.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:14.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:14.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:14.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:14.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:14.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:14.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:14.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:14.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:15.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:15.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:15.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:16.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:16.145 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:16.145 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:16.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724267 2024-04-25T08:20:16.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724267 '!=' null 2024-04-25T08:20:16.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:16.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:16.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:16.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:16.899 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:16.899 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:16.899 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:16.900 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:16.901 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:16.901 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:16.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:17.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:17.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:17.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:17.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:17.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:17.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:17.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:17.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:17.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:17.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:17.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:17.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:18.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:18.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:18.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:18.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712481 2024-04-25T08:20:18.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712481 '!=' null 2024-04-25T08:20:18.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:18.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:18.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:18.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:19.014 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:19.015 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:19.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:19.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:19.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:19.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:19.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:19.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:19.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:19.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:19.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:19.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:19.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:19.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:19.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:20.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:21.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850432 2024-04-25T08:20:21.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850432 '!=' null 2024-04-25T08:20:21.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:21.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:21.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:21.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:21.169 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:21.169 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:21.169 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:21.169 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:21.169 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [ 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "item": "3.0", 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "prio": 190, 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "can_preempt": true 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:21.171 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:21.171 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:21.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:21.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:21.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:21.285 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:21.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:21.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:21.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:21.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:21.286 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:21.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:21.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:21.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:21.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:22.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:22.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:22.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:22.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:22.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:22.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:22.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:22.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:22.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:23.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712423 2024-04-25T08:20:23.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712423 '!=' null 2024-04-25T08:20:23.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:23.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:23.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:23.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:23.272 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:23.273 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:23.273 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:23.273 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:23.273 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:23.273 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:23.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:23.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:23.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:23.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:23.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:23.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:23.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:23.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:23.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:23.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:23.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:23.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:23.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:24.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:24.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:24.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22275904 2024-04-25T08:20:24.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22275904 '!=' null 2024-04-25T08:20:24.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:24.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:24.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:24.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:25.073 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:25.074 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:25.074 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:25.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:25.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:25.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:25.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:25.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:25.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:25.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:25.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:25.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:25.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:20:25.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:25.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:25.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:26.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:20:26.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:20:26.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:26.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:26.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:26.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:26.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:26.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:26.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:26.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793097 2024-04-25T08:20:26.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793097 '!=' null 2024-04-25T08:20:26.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:26.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:26.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:26.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:26.853 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:26.853 INFO:tasks.workunit.client.0.smithi031.stdout: "local_reservations": { 2024-04-25T08:20:26.853 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "remote_reservations": { 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "max_allowed": 1, 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "min_priority": 0, 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "queues": [], 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: "in_progress": [] 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:26.854 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:26.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:26.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:26.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:26.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:26.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:26.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:26.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:26.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:26.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:27.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:20:27.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:27.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:27.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:27.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:20:27.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:20:27.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:20:27.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:20:28.385 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:20:28.386 INFO:tasks.workunit.client.0.smithi031.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:16:38.670211+0000 95'600 121:1144 [1,4] 1 [1,4] 1 0'0 2024-04-25T08:07:39.591918+0000 0'0 2024-04-25T08:07:39.591918+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:43:50.981057+0000 0 0 2024-04-25T08:20:28.386 INFO:tasks.workunit.client.0.smithi031.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:20:22.669463+0000 95'600 121:1174 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:07:27.554022+0000 0'0 2024-04-25T08:07:27.554022+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:59:20.938665+0000 0 0 2024-04-25T08:20:28.386 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:18:31.116574+0000 95'600 121:1190 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:07:21.890135+0000 0'0 2024-04-25T08:07:21.890135+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:35:39.698184+0000 0 0 2024-04-25T08:20:28.386 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:20:28.386 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:20:28.386 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202871/ceph-osd.1.asok 2024-04-25T08:20:28.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:20:28.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202871/ceph-osd.1.asok dump_pgstate_history 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout: "pgs": [ 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout: "pg": "7.0", 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout: "history": [ 2024-04-25T08:20:28.507 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "62", 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:39.914594+0000", 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:39.914613+0000" 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:39.914626+0000", 2024-04-25T08:20:28.508 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:40.078705+0000" 2024-04-25T08:20:28.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.509 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.509 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.509 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:40.078707+0000", 2024-04-25T08:20:28.509 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:40.078931+0000" 2024-04-25T08:20:28.509 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:40.078932+0000", 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:40.078960+0000" 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:40.078961+0000", 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:40.447333+0000" 2024-04-25T08:20:28.510 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:39.914619+0000", 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:40.447358+0000" 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:40.447797+0000", 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.101391+0000" 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.511 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.101393+0000", 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.101479+0000" 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.101480+0000", 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.965568+0000" 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:40.447359+0000", 2024-04-25T08:20:28.512 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.965612+0000" 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:39.914614+0000", 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.965709+0000" 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:39.914589+0000", 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.965722+0000" 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "63", 2024-04-25T08:20:28.513 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.965724+0000", 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.966319+0000" 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "93", 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.514 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.966331+0000", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:41.966363+0000" 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.966384+0000", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.093192+0000" 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.093194+0000", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.093296+0000" 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.093297+0000", 2024-04-25T08:20:28.515 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.093317+0000" 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.093318+0000", 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.409706+0000" 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.966374+0000", 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.409728+0000" 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.516 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.410038+0000", 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.958652+0000" 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.958653+0000", 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:42.958751+0000" 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.958751+0000", 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.604877+0000" 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.517 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:42.409729+0000", 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.604913+0000" 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.966364+0000", 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.604992+0000" 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:41.966320+0000", 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.604999+0000" 2024-04-25T08:20:28.518 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "93", 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605000+0000", 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.605419+0000" 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.519 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "100", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605435+0000", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.605460+0000" 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605476+0000", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.605717+0000" 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605718+0000", 2024-04-25T08:20:28.520 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.605855+0000" 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605856+0000", 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.605886+0000" 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605887+0000", 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:12.274709+0000" 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.521 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605468+0000", 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:12.274733+0000" 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:12.274899+0000", 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:12.342190+0000" 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.522 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:12.342192+0000", 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:12.342342+0000" 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:12.342343+0000", 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.293623+0000" 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:12.274734+0000", 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.293665+0000" 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.523 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605461+0000", 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.293880+0000" 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.605420+0000", 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.293889+0000" 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "101", 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.524 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.293890+0000", 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.294656+0000" 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "102", 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.294670+0000", 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.294701+0000" 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.525 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.294722+0000", 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.436348+0000" 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.436350+0000", 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.436521+0000" 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.294711+0000", 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.436541+0000" 2024-04-25T08:20:28.526 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.436542+0000", 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.695593+0000" 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.294702+0000", 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.695657+0000" 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.294657+0000", 2024-04-25T08:20:28.527 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.695668+0000" 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "102", 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.695669+0000", 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.696370+0000" 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "116", 2024-04-25T08:20:28.528 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.696383+0000", 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.696446+0000" 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.696466+0000", 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.823035+0000" 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.823037+0000", 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.823332+0000" 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.529 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.823334+0000", 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:57.823413+0000" 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.823414+0000", 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:58.137428+0000" 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.696457+0000", 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:58.137457+0000" 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.530 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:58.140626+0000", 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:58.325681+0000" 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:58.325683+0000", 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:58.326027+0000" 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:58.326028+0000", 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:13:58.326804+0000" 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.531 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:58.326806+0000", 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:28.623118+0000" 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:28.623119+0000", 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:28.623343+0000" 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:20:28.532 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:28.623343+0000", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:41.812781+0000" 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:41.812781+0000", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:41.813194+0000" 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:41.813194+0000", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:36.470570+0000" 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:16:36.470572+0000", 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:37.164028+0000" 2024-04-25T08:20:28.533 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:58.137458+0000", 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:37.164086+0000" 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.696447+0000", 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:37.164133+0000" 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:13:57.696370+0000", 2024-04-25T08:20:28.534 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:37.164150+0000" 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "116", 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:16:37.164151+0000", 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:37.164752+0000" 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.535 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "pg": "3.0", 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "history": [ 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "40", 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.493460+0000", 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:28.493479+0000" 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.536 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.493492+0000", 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:28.600326+0000" 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.600327+0000", 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:28.600483+0000" 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.600485+0000", 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:28.600510+0000" 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.537 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.600511+0000", 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:29.009902+0000" 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.493486+0000", 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:29.009927+0000" 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:29.010263+0000", 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.043178+0000" 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.538 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:29.009929+0000", 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.043213+0000" 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.493480+0000", 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.043253+0000" 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:28.493454+0000", 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.043261+0000" 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.539 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "40", 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.043262+0000", 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.043626+0000" 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "90", 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.540 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.043634+0000", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.043655+0000" 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.043687+0000", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.211142+0000" 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.211144+0000", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.211301+0000" 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.541 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.211302+0000", 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.211327+0000" 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.211328+0000", 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.620751+0000" 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.043664+0000", 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:30.620779+0000" 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.542 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.621287+0000", 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:31.161002+0000" 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:31.161004+0000", 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:31.161109+0000" 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:31.161110+0000", 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.534497+0000" 2024-04-25T08:20:28.543 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.620781+0000", 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.534521+0000" 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.043656+0000", 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.534575+0000" 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:30.043627+0000", 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.534580+0000" 2024-04-25T08:20:28.544 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "90", 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.534581+0000", 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.534834+0000" 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "108", 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.545 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.534840+0000", 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.534857+0000" 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.534867+0000", 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.535163+0000" 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.535164+0000", 2024-04-25T08:20:28.546 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.535244+0000" 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.535245+0000", 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.535263+0000" 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.535263+0000", 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.960052+0000" 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.534863+0000", 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:10.960073+0000" 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.547 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.960229+0000", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.028177+0000" 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.028178+0000", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:08:11.028260+0000" 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:11.028261+0000", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.054103+0000" 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.548 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.960075+0000", 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.054166+0000" 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.534857+0000", 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.054370+0000" 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:08:10.534835+0000", 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.054378+0000" 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.549 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "108", 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.054380+0000", 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.055015+0000" 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "110", 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.550 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.055026+0000", 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.055060+0000" 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.055081+0000", 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.056270+0000" 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.056271+0000", 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.056438+0000" 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.551 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.055072+0000", 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.056458+0000" 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.056459+0000", 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.558471+0000" 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.055061+0000", 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.558535+0000" 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.552 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.055016+0000", 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.558545+0000" 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "110", 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.553 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.558546+0000", 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.559048+0000" 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "120", 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.554 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.559059+0000", 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.559095+0000" 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.557 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.559117+0000", 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.817549+0000" 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.817552+0000", 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.817839+0000" 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.817840+0000", 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:13.817915+0000" 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.558 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.817916+0000", 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:14.110726+0000" 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.559105+0000", 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:14.110748+0000" 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:14.113481+0000", 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:14.371384+0000" 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.559 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:14.371385+0000", 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:28.623278+0000" 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:28.623278+0000", 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:28.623759+0000" 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:28.623760+0000", 2024-04-25T08:20:28.560 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:41.812599+0000" 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:41.812599+0000", 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:41.812828+0000" 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:41.812828+0000", 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.057672+0000" 2024-04-25T08:20:28.561 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:18:29.057672+0000", 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.058207+0000" 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:18:29.058207+0000", 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:20:21.781831+0000" 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:20:21.781832+0000", 2024-04-25T08:20:28.562 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:20:22.057753+0000" 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:14.110749+0000", 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:20:22.057780+0000" 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.559096+0000", 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:20:22.057819+0000" 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.563 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:13.559049+0000", 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:20:22.057826+0000" 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "120", 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.564 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:20:22.057827+0000", 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:20:22.058305+0000" 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "pg": "1.0", 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "history": [ 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "27", 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.565 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Initial", 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485098+0000", 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.485624+0000" 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "27", 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485625+0000", 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.485693+0000" 2024-04-25T08:20:28.566 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "29", 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485700+0000", 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.485720+0000" 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485746+0000", 2024-04-25T08:20:28.567 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.619003+0000" 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.619005+0000", 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.619198+0000" 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.619199+0000", 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.619223+0000" 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.619223+0000", 2024-04-25T08:20:28.568 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.807937+0000" 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485736+0000", 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:22.807952+0000" 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.808374+0000", 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:23.415907+0000" 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:23.415908+0000", 2024-04-25T08:20:28.569 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:23.415976+0000" 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:23.415976+0000", 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.022528+0000" 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.807955+0000", 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.022564+0000" 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.570 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485721+0000", 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.022652+0000" 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:22.485693+0000", 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.022659+0000" 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "29", 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.571 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.022660+0000", 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.023100+0000" 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "104", 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.023129+0000", 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.023174+0000" 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.572 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.023194+0000", 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.170885+0000" 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.170887+0000", 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.171060+0000" 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.171061+0000", 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.171090+0000" 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.573 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.171090+0000", 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.747338+0000" 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.023186+0000", 2024-04-25T08:20:28.574 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:24.747360+0000" 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.747655+0000", 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:25.351779+0000" 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.575 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:25.351781+0000", 2024-04-25T08:20:28.576 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:07:25.351944+0000" 2024-04-25T08:20:28.576 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.576 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.576 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:20:28.576 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:25.351945+0000", 2024-04-25T08:20:28.576 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.657972+0000" 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.747361+0000", 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.658065+0000" 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.023175+0000", 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.658267+0000" 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:07:24.023101+0000", 2024-04-25T08:20:28.577 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.658276+0000" 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "104", 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.658277+0000", 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.658966+0000" 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.578 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "106", 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.658977+0000", 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.659009+0000" 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.659030+0000", 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.801123+0000" 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.579 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.801125+0000", 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.801287+0000" 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.659021+0000", 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:06.801308+0000" 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.801308+0000", 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.344758+0000" 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.580 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.659009+0000", 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.344848+0000" 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:06.658967+0000", 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.344859+0000" 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "106", 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.581 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.344861+0000", 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.345355+0000" 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "118", 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Start", 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.345368+0000", 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.345400+0000" 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.582 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.345420+0000", 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.504456+0000" 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.504458+0000", 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.504651+0000" 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.504652+0000", 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.504691+0000" 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.583 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.504691+0000", 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.786543+0000" 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.345410+0000", 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:07.786572+0000" 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.789474+0000", 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:14:08.167541+0000" 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.584 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:08.167542+0000", 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:36.471007+0000" 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:16:36.471007+0000", 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:16:36.471744+0000" 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:20:28.585 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:16:36.471746+0000", 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.057199+0000" 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:18:29.057200+0000", 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.475398+0000" 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary/Active", 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.786573+0000", 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.475450+0000" 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.586 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started/Primary", 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.345401+0000", 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.475500+0000" 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Started", 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:14:07.345355+0000", 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.475509+0000" 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": "118", 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "states": [ 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "Reset", 2024-04-25T08:20:28.587 INFO:tasks.workunit.client.0.smithi031.stdout: "enter": "2024-04-25T08:18:29.475510+0000", 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: "exit": "2024-04-25T08:18:29.476140+0000" 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T08:20:28.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:20:28.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-04-25T08:20:28.589 INFO:tasks.workunit.client.0.smithi031.stdout:TEST PASSED 2024-04-25T08:20:28.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-04-25T08:20:28.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:20:28.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:20:29.055 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test1' does not exist 2024-04-25T08:20:29.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-04-25T08:20:29.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:20:29.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:20:29.880 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test3' does not exist 2024-04-25T08:20:29.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-04-25T08:20:29.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-04-25T08:20:29.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-04-25T08:20:30.439 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test7' does not exist 2024-04-25T08:20:30.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:20:30.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:20:30.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:20:30.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:20:30.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:20:30.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:20:31.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:20:31.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:20:31.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:20:31.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:20:31.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:20:31.781 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:20:31.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:20:31.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:20:31.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:20:31.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:20:31.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:20:31.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:20:31.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:20:31.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:20:31.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:20:31.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:20:31.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:20:31.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:20:31.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:20:31.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:31.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:20:31.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202871 2024-04-25T08:20:31.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:20:31.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-prio 0 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:20:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:20:31.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:20:31.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:20:31.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:20:31.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:20:31.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:20:31.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:20:31.874 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:20:31.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:20:31.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:20:31.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:20:31.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:20:31.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:20:31.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:20:31.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:20:31.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:20:31.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:31.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202871 2024-04-25T08:20:31.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202871 2024-04-25T08:20:31.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:20:31.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:20:31.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:20:31.880 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:20:31.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:20:31.892 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-04-25T08:20:31.892 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-04-25T08:20:31.942 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:20:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-space 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none ' 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-04-25T08:20:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-04-25T08:20:31.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-04-25T08:20:31.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:20:31.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-04-25T08:20:31.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:20:31.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:20:31.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:20:31.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:20:31.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:20:31.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:20:31.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:20:31.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:20:31.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:20:31.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:20:31.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:20:31.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:20:31.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:20:31.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:20:31.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:20:31.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:20:31.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:20:31.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:31.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:31.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257899 2024-04-25T08:20:31.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:20:31.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:20:31.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-04-25T08:20:31.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:20:31.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:31.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:31.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257899 2024-04-25T08:20:31.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:20:31.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:20:31.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local pools=1 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:91: TEST_recovery_test_simple: local OSDS=2 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-04-25T08:20:31.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-04-25T08:20:32.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257899/$cluster-$name.asok' 2024-04-25T08:20:32.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:20:32.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:20:32.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:20:32.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:20:32.657 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:20:32.657 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:20:32.657 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:20:32.657 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:20:32.657 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:20:32.657 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:20:32.658 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:20:32.658 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:32.658 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:32.658 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257899/ceph-mon.a.asok 2024-04-25T08:20:32.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:20:32.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.257899/ceph-mon.a.asok config get fsid 2024-04-25T08:20:32.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:20:32.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:20:32.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:20:32.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:20:32.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:20:32.778 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257899/ceph-mon.a.asok 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:20:32.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.257899/ceph-mon.a.asok config get mon_host 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-04-25T08:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:20:33.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:20:33.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:20:33.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:20:33.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:20:33.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:33.804 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:33.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257899/$cluster-$name.asok' 2024-04-25T08:20:33.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:20:33.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:20:33.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: export CEPH_ARGS 2024-04-25T08:20:33.841 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:20:33.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-04-25T08:20:33.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:33.847 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257899/$cluster-$name.asok' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-04-25T08:20:33.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:20:33.849 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 33863844-71eb-44f8-8cdf-03b1e9fcffac 2024-04-25T08:20:33.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33863844-71eb-44f8-8cdf-03b1e9fcffac 2024-04-25T08:20:33.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 33863844-71eb-44f8-8cdf-03b1e9fcffac' 2024-04-25T08:20:33.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:20:33.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBREipmAte3MxAA5eMVQIVTyGO9fxfzaXRGbA== 2024-04-25T08:20:33.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBREipmAte3MxAA5eMVQIVTyGO9fxfzaXRGbA=="}' 2024-04-25T08:20:33.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33863844-71eb-44f8-8cdf-03b1e9fcffac -i td/osd-recovery-space/0/new.json 2024-04-25T08:20:34.174 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:20:34.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-04-25T08:20:34.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBREipmAte3MxAA5eMVQIVTyGO9fxfzaXRGbA== --osd-uuid 33863844-71eb-44f8-8cdf-03b1e9fcffac 2024-04-25T08:20:34.228 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.227+0000 7f7cd37297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:34.262 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.263+0000 7f7cd37297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:34.263 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.263+0000 7f7cd37297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:34.264 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.263+0000 7f7cd37297c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:20:34.264 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.263+0000 7f7cd37297c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:20:34.264 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.263+0000 7f7cd37297c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:20:34.300 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:34.299+0000 7f7cd37297c0 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-04-25T08:20:41.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-04-25T08:20:41.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:20:41.076 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:20:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:20:41.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:20:41.646 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:20:41.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:20:41.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:20:41.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:20:41.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:20:41.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:20:41.681 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:41.679+0000 7fab85c6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:41.714 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:41.715+0000 7fab85c6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:41.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:41.715+0000 7fab85c6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:42.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:20:42.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:20:42.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:20:42.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:20:42.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:20:42.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:42.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:20:42.107 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:20:42.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:42.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:42.287 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:42.287+0000 7fab85c6b7c0 -1 Falling back to public interface 2024-04-25T08:20:42.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:43.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:43.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:43.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:20:43.539 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:20:43.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:43.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:43.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:44.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:44.977 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:20:44.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:20:44.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:44.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:45.045 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:20:45.047+0000 7fab85c6b7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:20:45.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:46.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:46.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:46.579 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:20:46.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:20:46.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:46.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:47.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:48.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:48.057 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:20:48.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:48.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:20:48.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:48.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:48.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:49.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:49.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:49.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:20:49.509 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:20:49.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:49.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:50.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:51.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:51.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:51.038 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:20:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:20:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:51.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:52.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:52.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:52.480 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:20:52.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:20:52.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:52.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:53.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:54.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:54.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:54.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:20:54.030 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:20:54.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:54.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:54.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:55.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:55.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:55.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:20:55.495 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:20:55.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:55.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:56.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:57.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:57.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:57.021 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:20:57.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:20:57.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:57.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:57.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:58.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:58.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:58.463 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:20:58.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:20:58.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:58.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:20:59.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:00.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:00.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:00.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:21:00.097 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:21:00.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:00.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:01.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:02.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:02.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:02.127 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:21:02.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:21:02.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:02.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:03.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:04.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:04.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:04.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:21:04.101 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:21:04.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:04.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:04.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:05.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:05.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:05.897 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:21:05.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:21:05.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:05.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:06.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:07.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:07.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:07.843 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:21:07.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:21:07.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:07.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:08.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:09.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:09.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:09.665 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:21:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:21:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:09.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:10.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:11.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:11.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:11.224 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:21:11.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:21:11.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:11.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:21:11.698 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1186599225,v1:127.0.0.1:6803/1186599225] [v2:127.0.0.1:6804/1186599225,v1:127.0.0.1:6805/1186599225] exists,up 33863844-71eb-44f8-8cdf-03b1e9fcffac 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:21:11.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:21:11.702 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257899/$cluster-$name.asok' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:21:11.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:21:11.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62e9576d-f090-4348-8d05-9a3d91e01ea2 2024-04-25T08:21:11.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62e9576d-f090-4348-8d05-9a3d91e01ea2' 2024-04-25T08:21:11.705 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 62e9576d-f090-4348-8d05-9a3d91e01ea2 2024-04-25T08:21:11.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:21:11.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3EipmxpRuKxAAQNRxdkSh3lV7q/HvykBdUQ== 2024-04-25T08:21:11.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3EipmxpRuKxAAQNRxdkSh3lV7q/HvykBdUQ=="}' 2024-04-25T08:21:11.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62e9576d-f090-4348-8d05-9a3d91e01ea2 -i td/osd-recovery-space/1/new.json 2024-04-25T08:21:12.228 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:21:12.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-04-25T08:21:12.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3EipmxpRuKxAAQNRxdkSh3lV7q/HvykBdUQ== --osd-uuid 62e9576d-f090-4348-8d05-9a3d91e01ea2 2024-04-25T08:21:12.282 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.283+0000 7f19d52e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:12.298 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.299+0000 7f19d52e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:12.299 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.299+0000 7f19d52e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:12.299 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.299+0000 7f19d52e17c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:21:12.299 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.299+0000 7f19d52e17c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:21:12.299 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.299+0000 7f19d52e17c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:21:12.337 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:12.339+0000 7f19d52e17c0 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-04-25T08:21:19.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-04-25T08:21:19.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:21:19.344 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:21:19.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:21:19.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:21:19.916 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:21:19.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:21:19.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:21:19.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:21:19.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dbc197fa-51a7-4ca2-a2c9-98df0a856dcd --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.257899/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:21:19.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:21:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:19.951+0000 7f75c17007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:19.962 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:19.963+0000 7f75c17007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:19.964 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:19.963+0000 7f75c17007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:20.331 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:21:20.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:21:20.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:20.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:20.534 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:20.535+0000 7f75c17007c0 -1 Falling back to public interface 2024-04-25T08:21:20.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:21.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:21.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:21.780 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:21:21.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:21:21.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:21.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:22.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:23.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:23.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:23.219 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:21:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:21:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:23.321 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:21:23.319+0000 7f75c17007c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:21:23.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:24.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:24.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:24.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:21:24.693 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:21:24.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:24.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:25.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:26.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:26.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:26.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:21:26.294 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:21:26.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:26.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:26.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:27.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:27.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:27.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:21:27.739 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:21:27.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:27.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:28.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:29.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:29.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:29.272 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:21:29.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:21:29.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:29.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:29.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:30.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:30.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:30.710 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:21:30.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:21:30.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:30.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:31.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:32.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:32.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:32.212 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:21:32.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:21:32.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:32.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:32.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:33.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:33.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:33.664 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:21:33.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:21:33.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:33.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:34.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:35.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:35.248 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:21:35.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:21:35.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:35.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:35.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:36.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:36.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:36.745 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:21:36.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:21:36.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:36.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:37.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:38.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:38.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:38.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:21:38.347 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:21:38.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:38.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:39.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:40.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:40.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:40.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:21:40.152 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:21:40.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:40.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:40.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:41.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:41.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:41.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:21:41.984 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:21:41.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:41.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:42.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:43.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:43.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:43.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:21:43.863 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:21:43.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:43.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:44.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:45.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:45.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:45.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:21:45.732 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:21:45.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:45.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:47.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:48.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:48.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:48.029 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:21:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:21:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:48.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:48.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:49.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:49.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:49.918 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:21:49.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:21:49.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:49.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:21:50.354 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2416599258,v1:127.0.0.1:6811/2416599258] [v2:127.0.0.1:6812/2416599258,v1:127.0.0.1:6813/2416599258] exists,up 62e9576d-f090-4348-8d05-9a3d91e01ea2 2024-04-25T08:21:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:21:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:21:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:21:50.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-04-25T08:21:51.172 INFO:tasks.workunit.client.0.smithi031.stderr:osd set-nearfull-ratio 0.4 2024-04-25T08:21:51.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-04-25T08:21:52.075 INFO:tasks.workunit.client.0.smithi031.stderr:osd set-backfillfull-ratio 0.45 2024-04-25T08:21:52.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:104: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-04-25T08:21:53.104 INFO:tasks.workunit.client.0.smithi031.stderr:osd set-full-ratio 0.5 2024-04-25T08:21:53.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:21:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:21:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: create_pool test1 1 1 2024-04-25T08:21:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:21:53.679 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test1' already exists 2024-04-25T08:21:53.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:21:54.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:109: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:21:55.707 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 1 2024-04-25T08:21:55.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: wait_for_clean 2024-04-25T08:21:55.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:21:55.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:21:55.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:21:55.733 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:21:55.734 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:21:55.734 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:21:55.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:21:55.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:21:55.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:21:55.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:21:55.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:21:55.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:21:55.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:21:55.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:21:55.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:21:56.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:21:56.350 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T08:21:56.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:21:56.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:21:56.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:21:56.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836498 2024-04-25T08:21:56.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836498 2024-04-25T08:21:56.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836498' 2024-04-25T08:21:56.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:21:56.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:21:56.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672966 2024-04-25T08:21:56.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672966 2024-04-25T08:21:56.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836498 1-42949672966' 2024-04-25T08:21:56.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:21:56.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836498 2024-04-25T08:21:56.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:21:56.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:21:56.749 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836498 2024-04-25T08:21:56.749 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:21:56.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836498 2024-04-25T08:21:56.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-25T08:21:56.750 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836498 2024-04-25T08:21:56.751 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:21:57.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-25T08:21:57.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:21:57.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672966 2024-04-25T08:21:57.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:21:57.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:21:57.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672966 2024-04-25T08:21:57.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:21:57.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672966 2024-04-25T08:21:57.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-04-25T08:21:57.213 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672966 2024-04-25T08:21:57.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:21:57.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672964 -lt 42949672966 2024-04-25T08:21:57.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:21:58.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:21:58.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:21:59.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672966 -lt 42949672966 2024-04-25T08:21:59.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:21:59.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:59.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:59.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:21:59.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:59.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:59.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:59.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:59.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:59.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:59.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:00.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:00.118 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:00.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:00.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:22:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:22:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:22:00.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-04-25T08:22:00.711 INFO:tasks.workunit.client.0.smithi031.stderr:5+0 records in 2024-04-25T08:22:00.711 INFO:tasks.workunit.client.0.smithi031.stderr:5+0 records out 2024-04-25T08:22:00.711 INFO:tasks.workunit.client.0.smithi031.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000258256 s, 19.8 MB/s 2024-04-25T08:22:00.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: seq 1 600 2024-04-25T08:22:00.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:00.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-04-25T08:22:00.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:00.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-04-25T08:22:01.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-04-25T08:22:01.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-04-25T08:22:01.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-04-25T08:22:01.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-04-25T08:22:01.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-04-25T08:22:01.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-04-25T08:22:01.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-04-25T08:22:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-04-25T08:22:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:01.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-04-25T08:22:02.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-04-25T08:22:02.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-04-25T08:22:02.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-04-25T08:22:02.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-04-25T08:22:02.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-04-25T08:22:02.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-04-25T08:22:02.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-04-25T08:22:02.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-04-25T08:22:02.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:02.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-04-25T08:22:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-04-25T08:22:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-04-25T08:22:03.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-04-25T08:22:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-04-25T08:22:03.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-04-25T08:22:03.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-04-25T08:22:03.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-04-25T08:22:03.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-04-25T08:22:03.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:03.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-04-25T08:22:04.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-04-25T08:22:04.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-04-25T08:22:04.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-04-25T08:22:04.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-04-25T08:22:04.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-04-25T08:22:04.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-04-25T08:22:04.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-04-25T08:22:04.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-04-25T08:22:04.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:04.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-04-25T08:22:05.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-04-25T08:22:05.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-04-25T08:22:05.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-04-25T08:22:05.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-04-25T08:22:05.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-04-25T08:22:05.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-04-25T08:22:05.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-04-25T08:22:05.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-04-25T08:22:05.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:05.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-04-25T08:22:06.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-04-25T08:22:06.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-04-25T08:22:06.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-04-25T08:22:06.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-04-25T08:22:06.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-04-25T08:22:06.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-04-25T08:22:06.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-04-25T08:22:06.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:06.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-04-25T08:22:07.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-04-25T08:22:07.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-04-25T08:22:07.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-04-25T08:22:07.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-04-25T08:22:07.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-04-25T08:22:07.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-04-25T08:22:07.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-04-25T08:22:07.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-04-25T08:22:07.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:07.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-04-25T08:22:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-04-25T08:22:08.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-04-25T08:22:08.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-04-25T08:22:08.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-04-25T08:22:08.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-04-25T08:22:08.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-04-25T08:22:08.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-04-25T08:22:08.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-04-25T08:22:08.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:08.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-04-25T08:22:09.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-04-25T08:22:09.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-04-25T08:22:09.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-04-25T08:22:09.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-04-25T08:22:09.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-04-25T08:22:09.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-04-25T08:22:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-04-25T08:22:09.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-04-25T08:22:09.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-04-25T08:22:09.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:09.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-04-25T08:22:10.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-04-25T08:22:10.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-04-25T08:22:10.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-04-25T08:22:10.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-04-25T08:22:10.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-04-25T08:22:10.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-04-25T08:22:10.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-04-25T08:22:10.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-04-25T08:22:10.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:10.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-04-25T08:22:11.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-04-25T08:22:11.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-04-25T08:22:11.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-04-25T08:22:11.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-04-25T08:22:11.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-04-25T08:22:11.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-04-25T08:22:11.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-04-25T08:22:11.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-04-25T08:22:11.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:11.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-04-25T08:22:12.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-04-25T08:22:12.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-04-25T08:22:12.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-04-25T08:22:12.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-04-25T08:22:12.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-04-25T08:22:12.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-04-25T08:22:12.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-04-25T08:22:12.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-04-25T08:22:12.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:12.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-04-25T08:22:13.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-04-25T08:22:13.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-04-25T08:22:13.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-04-25T08:22:13.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-04-25T08:22:13.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-04-25T08:22:13.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-04-25T08:22:13.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-04-25T08:22:13.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-04-25T08:22:13.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:13.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-04-25T08:22:14.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-04-25T08:22:14.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-04-25T08:22:14.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-04-25T08:22:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-04-25T08:22:14.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-04-25T08:22:14.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-04-25T08:22:14.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-04-25T08:22:14.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-04-25T08:22:14.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:14.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-04-25T08:22:15.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-04-25T08:22:15.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-04-25T08:22:15.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-04-25T08:22:15.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-04-25T08:22:15.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-04-25T08:22:15.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-04-25T08:22:15.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-04-25T08:22:15.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:15.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-04-25T08:22:16.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-04-25T08:22:16.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-04-25T08:22:16.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-04-25T08:22:16.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-04-25T08:22:16.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-04-25T08:22:16.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-04-25T08:22:16.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-04-25T08:22:16.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-04-25T08:22:16.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:16.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-04-25T08:22:17.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-04-25T08:22:17.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-04-25T08:22:17.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-04-25T08:22:17.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-04-25T08:22:17.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-04-25T08:22:17.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-04-25T08:22:17.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-04-25T08:22:17.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:17.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-04-25T08:22:18.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-04-25T08:22:18.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-04-25T08:22:18.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-04-25T08:22:18.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-04-25T08:22:18.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-04-25T08:22:18.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-04-25T08:22:18.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-04-25T08:22:18.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-04-25T08:22:18.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:18.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-04-25T08:22:19.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-04-25T08:22:19.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-04-25T08:22:19.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-04-25T08:22:19.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-04-25T08:22:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-04-25T08:22:19.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-04-25T08:22:19.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-04-25T08:22:19.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-04-25T08:22:19.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:19.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-04-25T08:22:20.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-04-25T08:22:20.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-04-25T08:22:20.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-04-25T08:22:20.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-04-25T08:22:20.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-04-25T08:22:20.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-04-25T08:22:20.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-04-25T08:22:20.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-04-25T08:22:20.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:20.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-04-25T08:22:21.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-04-25T08:22:21.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-04-25T08:22:21.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-04-25T08:22:21.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-04-25T08:22:21.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-04-25T08:22:21.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-04-25T08:22:21.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-04-25T08:22:21.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-04-25T08:22:21.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:21.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-04-25T08:22:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-04-25T08:22:22.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-04-25T08:22:22.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-04-25T08:22:22.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-04-25T08:22:22.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-04-25T08:22:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-04-25T08:22:22.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-04-25T08:22:22.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-04-25T08:22:22.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:22.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-04-25T08:22:23.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-04-25T08:22:23.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-04-25T08:22:23.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-04-25T08:22:23.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-04-25T08:22:23.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-04-25T08:22:23.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-04-25T08:22:23.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-04-25T08:22:23.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-04-25T08:22:23.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:23.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-04-25T08:22:24.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-04-25T08:22:24.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-04-25T08:22:24.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-04-25T08:22:24.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-04-25T08:22:24.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-04-25T08:22:24.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-04-25T08:22:24.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-04-25T08:22:24.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-04-25T08:22:24.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:24.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-04-25T08:22:25.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-04-25T08:22:25.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-04-25T08:22:25.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-04-25T08:22:25.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-04-25T08:22:25.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-04-25T08:22:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-04-25T08:22:25.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-04-25T08:22:25.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-04-25T08:22:25.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-04-25T08:22:25.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:25.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-04-25T08:22:26.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-04-25T08:22:26.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-04-25T08:22:26.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-04-25T08:22:26.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-04-25T08:22:26.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-04-25T08:22:26.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-04-25T08:22:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-04-25T08:22:26.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:26.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-04-25T08:22:27.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-04-25T08:22:27.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-04-25T08:22:27.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-04-25T08:22:27.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-04-25T08:22:27.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-04-25T08:22:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-04-25T08:22:27.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-04-25T08:22:27.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-04-25T08:22:27.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:27.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-04-25T08:22:28.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-04-25T08:22:28.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-04-25T08:22:28.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-04-25T08:22:28.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-04-25T08:22:28.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-04-25T08:22:28.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-04-25T08:22:28.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-04-25T08:22:28.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-04-25T08:22:28.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:28.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-04-25T08:22:29.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-04-25T08:22:29.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-04-25T08:22:29.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-04-25T08:22:29.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-04-25T08:22:29.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-04-25T08:22:29.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-04-25T08:22:29.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-04-25T08:22:29.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:29.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-04-25T08:22:30.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-04-25T08:22:30.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-04-25T08:22:30.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-04-25T08:22:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-04-25T08:22:30.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-04-25T08:22:30.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-04-25T08:22:30.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-04-25T08:22:30.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-04-25T08:22:30.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:30.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-04-25T08:22:31.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-04-25T08:22:31.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-04-25T08:22:31.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-04-25T08:22:31.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-04-25T08:22:31.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-04-25T08:22:31.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-04-25T08:22:31.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-04-25T08:22:31.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-04-25T08:22:31.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:31.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-04-25T08:22:32.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-04-25T08:22:32.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-04-25T08:22:32.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-04-25T08:22:32.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-04-25T08:22:32.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-04-25T08:22:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-04-25T08:22:32.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-04-25T08:22:32.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-04-25T08:22:32.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:32.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-04-25T08:22:33.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-04-25T08:22:33.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-04-25T08:22:33.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-04-25T08:22:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-04-25T08:22:33.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-04-25T08:22:33.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-04-25T08:22:33.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-04-25T08:22:33.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-04-25T08:22:33.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:33.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-04-25T08:22:34.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-04-25T08:22:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-04-25T08:22:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-04-25T08:22:34.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-04-25T08:22:34.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-04-25T08:22:34.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-04-25T08:22:34.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-04-25T08:22:34.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-04-25T08:22:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-04-25T08:22:35.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-04-25T08:22:35.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-04-25T08:22:35.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-04-25T08:22:35.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-04-25T08:22:35.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-04-25T08:22:35.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-04-25T08:22:35.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-04-25T08:22:35.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-04-25T08:22:35.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-04-25T08:22:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-04-25T08:22:36.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-04-25T08:22:36.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-04-25T08:22:36.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-04-25T08:22:36.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-04-25T08:22:36.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-04-25T08:22:36.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-04-25T08:22:36.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-04-25T08:22:36.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:36.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-04-25T08:22:37.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-04-25T08:22:37.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-04-25T08:22:37.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-04-25T08:22:37.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-04-25T08:22:37.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-04-25T08:22:37.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-04-25T08:22:37.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-04-25T08:22:37.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:37.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-04-25T08:22:38.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-04-25T08:22:38.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-04-25T08:22:38.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-04-25T08:22:38.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-04-25T08:22:38.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-04-25T08:22:38.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-04-25T08:22:38.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-04-25T08:22:38.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-04-25T08:22:38.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-04-25T08:22:38.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-04-25T08:22:38.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-04-25T08:22:38.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:38.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-04-25T08:22:39.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-04-25T08:22:39.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-04-25T08:22:39.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-04-25T08:22:39.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-04-25T08:22:39.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-04-25T08:22:39.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-04-25T08:22:39.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-04-25T08:22:39.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-04-25T08:22:39.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-04-25T08:22:39.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-04-25T08:22:39.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-04-25T08:22:39.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-04-25T08:22:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-04-25T08:22:39.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:39.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-04-25T08:22:40.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-04-25T08:22:40.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-04-25T08:22:40.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-04-25T08:22:40.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-04-25T08:22:40.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-04-25T08:22:40.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-04-25T08:22:40.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-04-25T08:22:40.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-04-25T08:22:40.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-04-25T08:22:40.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-04-25T08:22:40.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:40.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-04-25T08:22:41.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-04-25T08:22:41.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-04-25T08:22:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-04-25T08:22:41.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-04-25T08:22:41.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-04-25T08:22:41.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-04-25T08:22:41.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-04-25T08:22:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-04-25T08:22:41.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-04-25T08:22:41.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-04-25T08:22:41.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-04-25T08:22:41.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-04-25T08:22:41.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:41.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-04-25T08:22:42.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-04-25T08:22:42.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-04-25T08:22:42.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-04-25T08:22:42.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-04-25T08:22:42.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-04-25T08:22:42.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-04-25T08:22:42.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-04-25T08:22:42.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-04-25T08:22:42.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-04-25T08:22:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-04-25T08:22:42.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-04-25T08:22:42.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:42.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-04-25T08:22:43.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-04-25T08:22:43.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-04-25T08:22:43.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-04-25T08:22:43.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-04-25T08:22:43.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-04-25T08:22:43.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-04-25T08:22:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-04-25T08:22:43.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-04-25T08:22:43.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-04-25T08:22:43.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-04-25T08:22:43.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-04-25T08:22:43.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-04-25T08:22:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-04-25T08:22:44.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-04-25T08:22:44.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-04-25T08:22:44.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-04-25T08:22:44.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-04-25T08:22:44.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-04-25T08:22:44.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-04-25T08:22:44.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-04-25T08:22:44.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-04-25T08:22:44.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-04-25T08:22:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-04-25T08:22:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-04-25T08:22:44.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:44.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-04-25T08:22:45.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-04-25T08:22:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-04-25T08:22:45.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-04-25T08:22:45.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-04-25T08:22:45.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-04-25T08:22:45.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-04-25T08:22:45.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-04-25T08:22:45.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-04-25T08:22:45.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-04-25T08:22:45.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:45.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-04-25T08:22:46.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-04-25T08:22:46.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-04-25T08:22:46.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-04-25T08:22:46.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-04-25T08:22:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-04-25T08:22:46.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-04-25T08:22:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-04-25T08:22:46.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-04-25T08:22:46.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-04-25T08:22:46.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-04-25T08:22:46.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-04-25T08:22:46.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:46.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-04-25T08:22:47.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-04-25T08:22:47.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-04-25T08:22:47.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-04-25T08:22:47.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-04-25T08:22:47.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-04-25T08:22:47.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-04-25T08:22:47.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-04-25T08:22:47.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-04-25T08:22:47.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-04-25T08:22:47.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-04-25T08:22:47.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-04-25T08:22:47.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-04-25T08:22:47.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:47.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-04-25T08:22:48.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-04-25T08:22:48.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-04-25T08:22:48.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-04-25T08:22:48.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-04-25T08:22:48.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-04-25T08:22:48.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-04-25T08:22:48.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-04-25T08:22:48.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-04-25T08:22:48.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:48.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-04-25T08:22:49.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-04-25T08:22:49.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-04-25T08:22:49.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-04-25T08:22:49.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-04-25T08:22:49.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-04-25T08:22:49.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-04-25T08:22:49.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-04-25T08:22:49.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-04-25T08:22:49.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-04-25T08:22:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-04-25T08:22:49.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-04-25T08:22:49.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-04-25T08:22:49.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:49.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-04-25T08:22:50.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-04-25T08:22:50.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-04-25T08:22:50.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-04-25T08:22:50.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-04-25T08:22:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-04-25T08:22:50.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-04-25T08:22:50.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-04-25T08:22:50.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-04-25T08:22:50.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-04-25T08:22:50.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-04-25T08:22:50.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-04-25T08:22:50.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:50.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-04-25T08:22:51.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-04-25T08:22:51.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-04-25T08:22:51.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-04-25T08:22:51.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-04-25T08:22:51.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-04-25T08:22:51.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-04-25T08:22:51.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-04-25T08:22:51.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-04-25T08:22:51.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-04-25T08:22:51.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-04-25T08:22:51.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:51.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-04-25T08:22:52.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-04-25T08:22:52.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-04-25T08:22:52.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-04-25T08:22:52.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-04-25T08:22:52.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-04-25T08:22:52.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-04-25T08:22:52.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-04-25T08:22:52.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-04-25T08:22:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-04-25T08:22:52.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:52.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-04-25T08:22:53.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-04-25T08:22:53.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-04-25T08:22:53.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-04-25T08:22:53.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-04-25T08:22:53.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-04-25T08:22:53.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-04-25T08:22:53.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-04-25T08:22:53.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-04-25T08:22:53.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-04-25T08:22:53.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-04-25T08:22:53.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-04-25T08:22:53.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:53.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-04-25T08:22:54.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-04-25T08:22:54.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-04-25T08:22:54.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-04-25T08:22:54.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-04-25T08:22:54.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-04-25T08:22:54.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-04-25T08:22:54.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-04-25T08:22:54.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-04-25T08:22:54.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-04-25T08:22:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-04-25T08:22:54.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:54.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-04-25T08:22:55.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-04-25T08:22:55.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-04-25T08:22:55.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-04-25T08:22:55.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-04-25T08:22:55.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-04-25T08:22:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-04-25T08:22:55.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-04-25T08:22:55.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-04-25T08:22:55.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-04-25T08:22:55.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-04-25T08:22:55.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-04-25T08:22:55.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:55.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-04-25T08:22:56.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-04-25T08:22:56.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-04-25T08:22:56.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-04-25T08:22:56.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-04-25T08:22:56.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-04-25T08:22:56.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-04-25T08:22:56.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-04-25T08:22:56.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-04-25T08:22:56.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-04-25T08:22:56.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-04-25T08:22:56.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-04-25T08:22:56.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:56.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-04-25T08:22:57.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-04-25T08:22:57.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-04-25T08:22:57.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-04-25T08:22:57.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-04-25T08:22:57.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-04-25T08:22:57.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-04-25T08:22:57.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-04-25T08:22:57.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-04-25T08:22:57.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-04-25T08:22:57.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-04-25T08:22:57.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-04-25T08:22:57.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:57.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-04-25T08:22:58.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-04-25T08:22:58.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-04-25T08:22:58.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-04-25T08:22:58.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-04-25T08:22:58.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-04-25T08:22:58.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-04-25T08:22:58.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-04-25T08:22:58.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-04-25T08:22:58.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-04-25T08:22:58.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-04-25T08:22:58.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-04-25T08:22:58.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:58.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-04-25T08:22:59.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-04-25T08:22:59.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-04-25T08:22:59.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-04-25T08:22:59.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-04-25T08:22:59.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-04-25T08:22:59.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-04-25T08:22:59.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-04-25T08:22:59.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-04-25T08:22:59.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-04-25T08:22:59.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-04-25T08:22:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-04-25T08:22:59.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:22:59.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-04-25T08:23:00.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-04-25T08:23:00.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-04-25T08:23:00.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-04-25T08:23:00.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-04-25T08:23:00.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-04-25T08:23:00.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-04-25T08:23:00.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:23:00.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-04-25T08:23:00.580 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:23:00.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:23:00.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:23:00.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:23:00.741 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T08:23:00.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:23:00.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:23:00.901 INFO:tasks.workunit.client.0.smithi031.stdout:{} 2024-04-25T08:23:00.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: sleep 5 2024-04-25T08:23:05.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:23:06.334 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:23:06.335 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 591 0 0 0 0 3025920 0 0 591 0 591 active+clean 2024-04-25T08:21:56.245263+0000 23'591 23:608 [1] 1 [1] 1 0'0 2024-04-25T08:21:53.507176+0000 0'0 2024-04-25T08:21:53.507176+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:49:03.890671+0000 0 0 2024-04-25T08:23:06.335 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:23:06.335 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:23:06.335 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:23:06.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:23:06.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:23:06.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-04-25T08:23:07.123 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T08:23:07.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2024-04-25T08:23:07.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=30 2024-04-25T08:23:07.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2024-04-25T08:23:07.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-04-25T08:23:07.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-04-25T08:23:07.145 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 30 5 2024-04-25T08:23:07.146 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:23:07.146 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:23:07.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:23:07.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:23:07.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:23:07.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15') 2024-04-25T08:23:07.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-04-25T08:23:07.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-04-25T08:23:07.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-04-25T08:23:07.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:23:07.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:23:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:23:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T08:23:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:23:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:07.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:23:07.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836514 2024-04-25T08:23:07.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836514 2024-04-25T08:23:07.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836514' 2024-04-25T08:23:07.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:07.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:23:08.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-04-25T08:23:08.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-04-25T08:23:08.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836514 1-42949672982' 2024-04-25T08:23:08.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:08.044 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836514 2024-04-25T08:23:08.044 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:08.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:23:08.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836514 2024-04-25T08:23:08.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:08.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836514 2024-04-25T08:23:08.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-04-25T08:23:08.048 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836514 2024-04-25T08:23:08.048 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:23:08.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836514 2024-04-25T08:23:08.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:23:09.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:23:09.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:23:09.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836514 2024-04-25T08:23:09.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:09.963 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-04-25T08:23:09.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:09.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:23:09.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-04-25T08:23:09.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:09.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-04-25T08:23:09.968 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672982 2024-04-25T08:23:09.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-25T08:23:09.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:23:10.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-25T08:23:10.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-04-25T08:23:10.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:10.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:10.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-04-25T08:23:10.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-04-25T08:23:10.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-04-25T08:23:10.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-04-25T08:23:10.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-04-25T08:23:10.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-04-25T08:23:10.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-04-25T08:23:10.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-04-25T08:23:11.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-04-25T08:23:11.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-04-25T08:23:11.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-04-25T08:23:11.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-04-25T08:23:11.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ERRORS=0 2024-04-25T08:23:11.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:23:11.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: grep +recovery_toofull 2024-04-25T08:23:11.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: wc -l 2024-04-25T08:23:11.785 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:23:11.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-04-25T08:23:11.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:23:12.178 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:23:12.178 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:23:07.864911+0000 23'600 27:628 [1,0] 1 [1] 1 0'0 2024-04-25T08:21:53.507176+0000 0'0 2024-04-25T08:21:53.507176+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:23:12.178 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:23:12.178 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:23:12.178 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:23:12.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status 2024-04-25T08:23:12.760 INFO:tasks.workunit.client.0.smithi031.stdout: cluster: 2024-04-25T08:23:12.760 INFO:tasks.workunit.client.0.smithi031.stdout: id: dbc197fa-51a7-4ca2-a2c9-98df0a856dcd 2024-04-25T08:23:12.760 INFO:tasks.workunit.client.0.smithi031.stdout: health: HEALTH_ERR 2024-04-25T08:23:12.760 INFO:tasks.workunit.client.0.smithi031.stdout: 2 full osd(s) 2024-04-25T08:23:12.760 INFO:tasks.workunit.client.0.smithi031.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-04-25T08:23:12.760 INFO:tasks.workunit.client.0.smithi031.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: 1 pool(s) do not have an application enabled 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: 1 pool(s) full 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: OSD count 2 < osd_pool_default_size 3 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: services: 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: mon: 1 daemons, quorum a (age 2m) 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: mgr: x(active, since 2m) 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: osd: 2 osds: 2 up (since 83s), 2 in (since 2m); 1 remapped pgs 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: data: 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: pools: 1 pools, 1 pgs 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: objects: 600 objects, 2.9 MiB 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: usage: 5.9 MiB used, 1.2 MiB / 7.0 MiB avail 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:23:12.761 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:23:12.762 INFO:tasks.workunit.client.0.smithi031.stdout: io: 2024-04-25T08:23:12.762 INFO:tasks.workunit.client.0.smithi031.stdout: client: 7.5 KiB/s wr, 0 op/s rd, 1 op/s wr 2024-04-25T08:23:12.762 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T08:23:12.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-04-25T08:23:13.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-04-25T08:23:13.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-04-25T08:23:13.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-04-25T08:23:13.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:148: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-04-25T08:23:13.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-04-25T08:23:13.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-04-25T08:23:13.394 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-04-25T08:23:13.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-04-25T08:23:13.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:157: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-04-25T08:23:13.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:159: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-04-25T08:23:13.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:23:13.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-04-25T08:23:13.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:166: TEST_recovery_test_simple: delete_pool test1 2024-04-25T08:23:13.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:23:13.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:23:13.911 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test1' does not exist 2024-04-25T08:23:13.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:168: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-04-25T08:23:13.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:23:13.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:23:13.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:23:13.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:23:13.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:23:19.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:23:19.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-04-25T08:23:19.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:23:19.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:23:19.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:23:19.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:23:19.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:23:19.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:23:19.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:23:19.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:23:19.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:23:19.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:23:19.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:23:19.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:23:19.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:23:19.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:23:19.260 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:23:19.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:23:19.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:23:19.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:23:19.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:23:19.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:23:19.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:23:19.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:23:19.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:23:19.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:19.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:23:19.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257899 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-space 0 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:23:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:23:19.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:23:19.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:23:19.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:23:19.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:23:19.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:23:19.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:23:19.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:23:19.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:23:19.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:23:19.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:23:19.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:23:19.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:23:19.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:23:19.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:23:19.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:23:19.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:23:19.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:19.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257899 2024-04-25T08:23:19.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257899 2024-04-25T08:23:19.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:23:19.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:23:19.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:23:19.311 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:23:19.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:23:19.322 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-04-25T08:23:19.322 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-04-25T08:23:19.375 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:23:19.382 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:23:19.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:23:19.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:23:19.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:23:19.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-stats 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-04-25T08:23:19.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-04-25T08:23:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none ' 2024-04-25T08:23:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T08:23:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:23:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-04-25T08:23:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-04-25T08:23:19.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-04-25T08:23:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-04-25T08:23:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-04-25T08:23:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-04-25T08:23:19.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-04-25T08:23:19.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:23:19.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-04-25T08:23:19.387 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_erasure_remapped 2024-04-25T08:23:19.387 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_multi 2024-04-25T08:23:19.387 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_replicated_out1 2024-04-25T08:23:19.387 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_sizedown 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_sizeup 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_recovery_undersized' 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:23:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:23:19.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:23:19.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:23:19.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:23:19.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:23:19.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:23:19.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:23:19.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:23:19.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:23:19.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:23:19.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:23:19.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:23:19.394 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:23:19.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:23:19.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:23:19.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:23:19.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:23:19.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:19.398 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:19.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:23:19.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:23:19.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:23:19.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:23:19.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:23:19.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:19.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:19.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:23:19.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-04-25T08:23:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-04-25T08:23:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:23:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:23:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-04-25T08:23:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:23:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:23:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:23:19.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:23:19.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:23:19.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:23:19.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:23:19.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:23:19.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:23:19.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:23:19.920 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:23:19.920 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:19.920 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:19.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:23:19.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:23:20.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:23:20.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:23:20.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:23:20.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:23:20.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:23:20.142 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:23:20.142 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:23:20.142 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:23:20.142 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:23:20.142 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:23:20.143 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:20.143 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:20.143 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:23:20.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:23:20.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:23:20.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:23:20.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:23:20.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:23:20.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:23:20.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:23:20.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:23:20.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:23:21.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:23:21.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:23:21.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:23:21.306 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:23:21.306 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:21.306 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:21.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:23:21.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:23:21.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:23:21.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:23:21.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:23:21.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:23:21.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:23:21.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:23:21.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f55a504-f001-421c-910a-0865a45d91e0 2024-04-25T08:23:21.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8f55a504-f001-421c-910a-0865a45d91e0' 2024-04-25T08:23:21.347 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 8f55a504-f001-421c-910a-0865a45d91e0 2024-04-25T08:23:21.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:23:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5EipmKYXEFRAABhSOnc14QY12ekFenTsu2A== 2024-04-25T08:23:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5EipmKYXEFRAABhSOnc14QY12ekFenTsu2A=="}' 2024-04-25T08:23:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f55a504-f001-421c-910a-0865a45d91e0 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:23:21.725 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:23:21.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:23:21.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5EipmKYXEFRAABhSOnc14QY12ekFenTsu2A== --osd-uuid 8f55a504-f001-421c-910a-0865a45d91e0 2024-04-25T08:23:21.777 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.777+0000 7fa9e01c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:21.802 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.801+0000 7fa9e01c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:21.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.801+0000 7fa9e01c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:21.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.801+0000 7fa9e01c07c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:23:21.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.801+0000 7fa9e01c07c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:23:21.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.801+0000 7fa9e01c07c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:23:21.842 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:21.841+0000 7fa9e01c07c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:23:28.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:23:28.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:23:28.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:23:28.474 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:23:28.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:23:29.040 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:23:29.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:23:29.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:23:29.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:23:29.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:23:29.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:23:29.073 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:29.069+0000 7f4d7d7917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:29.107 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:29.105+0000 7f4d7d7917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:29.109 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:29.105+0000 7f4d7d7917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:29.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:23:29.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:23:29.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:23:29.479 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:23:29.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:23:29.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:23:29.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:29.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:23:29.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:29.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:29.680 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:29.677+0000 7f4d7d7917c0 -1 Falling back to public interface 2024-04-25T08:23:29.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:30.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:30.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:30.911 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:23:30.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:23:30.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:30.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:31.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:32.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:32.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:32.349 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:23:32.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:23:32.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:32.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:32.358 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:32.357+0000 7f4d7d7917c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:23:32.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:33.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:33.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:33.852 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:23:33.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:23:33.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:33.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:34.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:35.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:35.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:35.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:23:35.309 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:23:35.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:35.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:35.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:36.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:36.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:36.743 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:23:36.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:23:36.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:36.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:37.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:38.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:38.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:38.310 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:23:38.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:23:38.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:38.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:38.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:39.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:39.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:39.751 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:23:39.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:23:39.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:39.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:40.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:41.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:41.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:41.303 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:23:41.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:23:41.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:41.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:41.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:42.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:42.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:42.748 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:23:42.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:23:42.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:42.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:43.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:44.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:44.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:44.378 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:23:44.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:23:44.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:44.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:44.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:45.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:45.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:45.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:23:45.814 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:23:45.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:45.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:47.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:48.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:48.141 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:23:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:23:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:48.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:49.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:50.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:50.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:50.265 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:23:50.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:23:50.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:50.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:50.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:51.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:51.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:51.870 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:23:51.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:23:51.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:51.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:52.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:53.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:53.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:53.617 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:23:53.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:23:53.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:53.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:54.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:55.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:55.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:55.940 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:23:55.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:23:55.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:55.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:56.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:57.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:57.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:57.936 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:23:57.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:23:57.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:57.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:23:58.366 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3333353426,v1:127.0.0.1:6803/3333353426] [v2:127.0.0.1:6804/3333353426,v1:127.0.0.1:6805/3333353426] exists,up 8f55a504-f001-421c-910a-0865a45d91e0 2024-04-25T08:23:58.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:23:58.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:23:58.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:23:58.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:23:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65957e54-39da-4720-b414-9c0efd0922f0 2024-04-25T08:23:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 65957e54-39da-4720-b414-9c0efd0922f0' 2024-04-25T08:23:58.374 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 65957e54-39da-4720-b414-9c0efd0922f0 2024-04-25T08:23:58.374 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:23:58.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeEypmtHk+FxAABm/iQZPQ66nP9bF2iizvkg== 2024-04-25T08:23:58.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeEypmtHk+FxAABm/iQZPQ66nP9bF2iizvkg=="}' 2024-04-25T08:23:58.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65957e54-39da-4720-b414-9c0efd0922f0 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:23:58.927 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:23:58.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:23:58.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeEypmtHk+FxAABm/iQZPQ66nP9bF2iizvkg== --osd-uuid 65957e54-39da-4720-b414-9c0efd0922f0 2024-04-25T08:23:58.979 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:58.976+0000 7ff58c7557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:59.018 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:59.016+0000 7ff58c7557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:59.020 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:59.016+0000 7ff58c7557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:59.020 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:59.016+0000 7ff58c7557c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:23:59.020 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:59.016+0000 7ff58c7557c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:23:59.021 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:59.016+0000 7ff58c7557c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:23:59.065 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:23:59.060+0000 7ff58c7557c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:24:06.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:24:06.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:24:06.100 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:24:06.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:24:06.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:24:06.715 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:24:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:24:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:24:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:24:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:24:06.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:24:06.747 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:06.744+0000 7f10cef397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:06.779 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:06.776+0000 7f10cef397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:06.781 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:06.780+0000 7f10cef397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:07.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:24:07.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:24:07.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:24:07.165 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:24:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:24:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:24:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:24:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:07.355 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:07.352+0000 7f10cef397c0 -1 Falling back to public interface 2024-04-25T08:24:07.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:08.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:08.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:08.603 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:24:08.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:24:08.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:08.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:09.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:10.078 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:10.076+0000 7f10cef397c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:24:10.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:10.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:10.090 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:24:10.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:24:10.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:10.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:10.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:11.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:11.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:11.688 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:24:11.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:24:11.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:11.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:12.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:13.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:13.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:13.243 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:24:13.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:24:13.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:13.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:13.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:14.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:14.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:14.705 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:24:14.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:24:14.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:14.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:15.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:16.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:16.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:16.187 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:24:16.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:24:16.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:16.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:16.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:17.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:17.734 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:24:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:24:17.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:17.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:18.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:19.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:19.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:19.179 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:24:19.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:24:19.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:19.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:19.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:20.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:20.664 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:24:20.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:20.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:24:20.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:20.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:21.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:22.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:22.192 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:24:22.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:24:22.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:22.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:22.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:23.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:23.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:23.720 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:24:23.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:24:23.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:23.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:24.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:25.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:25.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:25.236 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:24:25.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:24:25.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:25.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:26.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:27.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:27.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:27.591 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:24:27.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:24:27.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:27.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:28.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:29.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:29.351 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:24:29.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:29.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:24:29.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:29.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:30.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:31.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:31.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:31.080 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:24:31.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:24:31.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:31.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:32.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:33.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:24:33.213 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:24:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:34.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:35.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:35.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:35.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:24:35.123 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:24:35.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:35.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:35.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:36.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:36.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:36.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:24:36.721 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:24:36.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:36.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:37.148 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3128734487,v1:127.0.0.1:6811/3128734487] [v2:127.0.0.1:6812/3128734487,v1:127.0.0.1:6813/3128734487] exists,up 65957e54-39da-4720-b414-9c0efd0922f0 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:24:37.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:24:37.153 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:24:37.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:24:37.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:24:37.155 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:24:37.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=137a10da-963a-4cad-b048-b3e2d1cabff7 2024-04-25T08:24:37.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 137a10da-963a-4cad-b048-b3e2d1cabff7' 2024-04-25T08:24:37.156 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 137a10da-963a-4cad-b048-b3e2d1cabff7 2024-04-25T08:24:37.156 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:24:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFEypmAlqoChAAQS3vCzJbjHRaPvKCBlCitA== 2024-04-25T08:24:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFEypmAlqoChAAQS3vCzJbjHRaPvKCBlCitA=="}' 2024-04-25T08:24:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 137a10da-963a-4cad-b048-b3e2d1cabff7 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:24:37.690 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:24:37.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:24:37.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFEypmAlqoChAAQS3vCzJbjHRaPvKCBlCitA== --osd-uuid 137a10da-963a-4cad-b048-b3e2d1cabff7 2024-04-25T08:24:37.743 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.744+0000 7f9916f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:37.786 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.788+0000 7f9916f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.788+0000 7f9916f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:37.788 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.788+0000 7f9916f7d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:24:37.788 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.788+0000 7f9916f7d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:24:37.788 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.788+0000 7f9916f7d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:24:37.825 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:37.828+0000 7f9916f7d7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:24:44.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:24:44.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:24:44.506 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:24:44.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:24:44.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:24:45.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:24:45.091 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:24:45.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:24:45.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:24:45.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:24:45.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:24:45.118 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:45.120+0000 7f54d0f5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:45.157 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:45.160+0000 7f54d0f5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:45.159 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:45.160+0000 7f54d0f5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:45.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:24:45.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:24:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:24:45.541 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:24:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:24:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:24:45.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:45.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:24:45.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:45.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:45.739 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:45.740+0000 7f54d0f5a7c0 -1 Falling back to public interface 2024-04-25T08:24:46.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:47.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:47.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:47.004 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:24:47.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:24:47.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:47.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:47.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:48.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:48.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:48.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:24:48.475 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:24:48.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:48.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:48.585 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:24:48.584+0000 7f54d0f5a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:24:48.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:49.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:49.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:49.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:24:49.959 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:24:49.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:49.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:50.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:51.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:51.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:51.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:24:51.493 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:24:51.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:51.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:52.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:53.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:53.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:53.156 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:24:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:24:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:53.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:54.641 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:24:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:24:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:56.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:56.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:56.135 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:24:56.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:24:56.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:56.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:57.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:57.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:57.692 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:24:57.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:24:57.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:57.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:58.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:59.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:59.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:59.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:24:59.134 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:24:59.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:59.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:24:59.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:00.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:00.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:00.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:25:00.690 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:25:00.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:00.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:01.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:02.132 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:25:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:25:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:02.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:02.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:03.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:03.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:03.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:25:03.674 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:25:03.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:03.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:04.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:05.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:05.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:05.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:25:05.313 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:25:05.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:05.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:06.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:07.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:07.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:07.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:25:07.156 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:25:07.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:07.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:07.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:08.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:08.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:08.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:25:08.818 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:25:08.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:08.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:09.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:10.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:10.924 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:25:10.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:10.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:25:10.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:10.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:11.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:12.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:12.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:12.866 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:25:12.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:25:12.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:12.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:13.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:14.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:14.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:14.480 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:25:14.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:25:14.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:14.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:15.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:16.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:16.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:16.333 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T08:25:16.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:25:16.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:16.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:25:16.791 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/303626509,v1:127.0.0.1:6819/303626509] [v2:127.0.0.1:6820/303626509,v1:127.0.0.1:6821/303626509] exists,up 137a10da-963a-4cad-b048-b3e2d1cabff7 2024-04-25T08:25:16.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:25:16.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:25:16.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:25:16.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:25:16.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:25:16.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:25:16.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:25:16.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:25:16.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:25:16.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:25:16.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:25:16.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:25:16.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a7a043a-a814-4e89-a8e5-10132abee05d 2024-04-25T08:25:16.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3a7a043a-a814-4e89-a8e5-10132abee05d' 2024-04-25T08:25:16.799 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 3a7a043a-a814-4e89-a8e5-10132abee05d 2024-04-25T08:25:16.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:25:16.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsEypm4oW8MBAARigYliTD5Pqmw9WivQNTHg== 2024-04-25T08:25:16.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsEypm4oW8MBAARigYliTD5Pqmw9WivQNTHg=="}' 2024-04-25T08:25:16.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a7a043a-a814-4e89-a8e5-10132abee05d -i td/osd-recovery-stats/3/new.json 2024-04-25T08:25:17.327 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:25:17.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:25:17.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsEypm4oW8MBAARigYliTD5Pqmw9WivQNTHg== --osd-uuid 3a7a043a-a814-4e89-a8e5-10132abee05d 2024-04-25T08:25:17.383 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.383+0000 7f9ef9c327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:17.422 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.423+0000 7f9ef9c327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:17.423 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.423+0000 7f9ef9c327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:17.423 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.423+0000 7f9ef9c327c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:25:17.424 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.423+0000 7f9ef9c327c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:25:17.424 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.423+0000 7f9ef9c327c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:25:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:17.459+0000 7f9ef9c327c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:25:24.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:25:24.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:25:24.348 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:25:24.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:25:24.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:25:24.996 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:25:24.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:25:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:25:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:25:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:25:24.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:25:25.030 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:25.031+0000 7fd4946c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:25.051 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:25.051+0000 7fd4946c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:25.052 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:25.051+0000 7fd4946c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:25.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:25:25.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:25:25.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:25:25.464 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:25:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:25:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:25:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:25:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:25.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:25.622 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:25.623+0000 7fd4946c27c0 -1 Falling back to public interface 2024-04-25T08:25:25.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:26.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:26.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:26.902 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:25:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:25:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:27.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:28.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:28.335 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:25:28.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:25:28.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:28.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:28.389 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:28.387+0000 7fd4946c27c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:25:28.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:29.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:29.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:29.779 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:25:29.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:25:29.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:29.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:30.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:31.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:31.458 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:25:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:25:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:31.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:31.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:32.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:32.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:32.977 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:25:32.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:25:32.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:32.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:33.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:34.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:34.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:34.503 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:25:34.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:25:34.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:34.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:35.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:36.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:36.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:36.133 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:25:36.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:25:36.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:36.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:36.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:37.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:37.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:37.779 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:38.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:39.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:39.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:39.426 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:25:39.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:25:39.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:39.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:39.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:40.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:40.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:40.892 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:25:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:25:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:41.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:42.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:42.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:42.427 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:25:42.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:25:42.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:42.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:43.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:44.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:44.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:44.009 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:25:44.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:25:44.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:44.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:44.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:45.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:45.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:45.949 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:25:45.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:25:45.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:45.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:46.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:47.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:47.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:47.746 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:25:47.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:25:47.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:47.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:48.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:49.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:49.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:49.501 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:25:49.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:25:49.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:49.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:50.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:51.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:51.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:51.391 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:25:51.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:25:51.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:51.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:52.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:53.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:53.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:53.240 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:25:53.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:25:53.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:53.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:54.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:55.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:55.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:55.121 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:25:55.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:25:55.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:55.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:25:55.707 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1674297203,v1:127.0.0.1:6827/1674297203] [v2:127.0.0.1:6828/1674297203,v1:127.0.0.1:6829/1674297203] exists,up 3a7a043a-a814-4e89-a8e5-10132abee05d 2024-04-25T08:25:55.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:25:55.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:25:55.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:25:55.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:25:55.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:25:55.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:25:55.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:25:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:25:55.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:25:55.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:25:55.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:25:55.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:25:55.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:25:55.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da62fea2-6250-4017-bf9f-0ff4956d174d 2024-04-25T08:25:55.715 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 da62fea2-6250-4017-bf9f-0ff4956d174d 2024-04-25T08:25:55.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 da62fea2-6250-4017-bf9f-0ff4956d174d' 2024-04-25T08:25:55.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:25:55.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTEypmKuIDLBAA1IYDIE/po0SUvkgO/SUHGQ== 2024-04-25T08:25:55.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTEypmKuIDLBAA1IYDIE/po0SUvkgO/SUHGQ=="}' 2024-04-25T08:25:55.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da62fea2-6250-4017-bf9f-0ff4956d174d -i td/osd-recovery-stats/4/new.json 2024-04-25T08:25:56.240 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:25:56.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:25:56.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTEypmKuIDLBAA1IYDIE/po0SUvkgO/SUHGQ== --osd-uuid da62fea2-6250-4017-bf9f-0ff4956d174d 2024-04-25T08:25:56.295 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.295+0000 7f576c7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:56.424 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.423+0000 7f576c7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:56.425 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.427+0000 7f576c7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:56.426 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.427+0000 7f576c7b37c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:25:56.426 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.427+0000 7f576c7b37c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:25:56.426 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.427+0000 7f576c7b37c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:25:56.542 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:25:56.543+0000 7f576c7b37c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:26:03.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:26:03.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:26:03.605 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T08:26:03.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:26:03.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:26:04.187 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T08:26:04.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:26:04.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:26:04.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:26:04.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:26:04.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:26:04.221 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:04.218+0000 7f981edb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:04.258 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:04.258+0000 7f981edb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:04.259 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:04.258+0000 7f981edb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:04.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:26:04.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:26:04.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:26:04.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:26:04.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:26:04.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:04.717 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:26:04.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:26:04.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:04.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:04.830 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:04.830+0000 7f981edb17c0 -1 Falling back to public interface 2024-04-25T08:26:05.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:06.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:06.176 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:26:06.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:06.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:26:06.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:06.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:06.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:07.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:07.474+0000 7f981edb17c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:26:07.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:07.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:07.669 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:26:07.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:26:07.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:07.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:08.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:09.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:09.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:09.246 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:26:09.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:26:09.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:09.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:09.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:10.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:10.798 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:26:10.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:10.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:26:10.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:10.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:11.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:12.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:12.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:12.366 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:26:12.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:26:12.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:12.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:12.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:13.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:13.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:13.860 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:26:13.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:26:13.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:13.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:14.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:15.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:15.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:15.457 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:26:15.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:26:15.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:15.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:15.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:16.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:16.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:16.932 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:26:16.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:26:16.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:16.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:17.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:18.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:18.521 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:26:18.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:26:18.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:18.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:20.028 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:26:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:26:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:20.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:20.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:21.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:21.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:21.652 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:26:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:26:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:22.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:23.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:23.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:23.302 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:26:23.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:26:23.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:23.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:24.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:25.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:25.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:25.408 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:26:25.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:26:25.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:25.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:26.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:27.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:27.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:27.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:26:27.155 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:26:27.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:27.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:28.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:29.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:29.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:29.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:26:29.154 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:26:29.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:29.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:30.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:31.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:31.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:31.141 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:26:31.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:26:31.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:31.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:31.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:32.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:32.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:32.991 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:26:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:26:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:32.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:33.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:34.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:34.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:34.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:26:34.722 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:26:34.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:34.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:26:35.210 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/347324553,v1:127.0.0.1:6835/347324553] [v2:127.0.0.1:6836/347324553,v1:127.0.0.1:6837/347324553] exists,up da62fea2-6250-4017-bf9f-0ff4956d174d 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:26:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:26:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:35.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:26:35.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:26:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2bfcfbe5-30ce-4428-84e0-959648658418 2024-04-25T08:26:35.217 INFO:tasks.workunit.client.0.smithi031.stdout:add osd5 2bfcfbe5-30ce-4428-84e0-959648658418 2024-04-25T08:26:35.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 2bfcfbe5-30ce-4428-84e0-959648658418' 2024-04-25T08:26:35.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:26:35.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7EypmIjKIDhAAVBfTqIc3KheC0fJ5PFOVZA== 2024-04-25T08:26:35.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7EypmIjKIDhAAVBfTqIc3KheC0fJ5PFOVZA=="}' 2024-04-25T08:26:35.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2bfcfbe5-30ce-4428-84e0-959648658418 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:26:35.859 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:26:35.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:26:35.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7EypmIjKIDhAAVBfTqIc3KheC0fJ5PFOVZA== --osd-uuid 2bfcfbe5-30ce-4428-84e0-959648658418 2024-04-25T08:26:35.918 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.918+0000 7f598d6757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:35.921 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.922+0000 7f598d6757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:35.922 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.922+0000 7f598d6757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:35.923 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.922+0000 7f598d6757c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:26:35.923 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.922+0000 7f598d6757c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:26:35.923 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.922+0000 7f598d6757c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:26:35.998 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:35.998+0000 7f598d6757c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:26:43.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:26:43.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:26:43.048 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd5 key to auth repository 2024-04-25T08:26:43.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:26:43.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:26:43.627 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.5 2024-04-25T08:26:43.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:26:43.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:26:43.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:26:43.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:26:43.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:26:43.660 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:43.658+0000 7fa3c72fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:43.691 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:43.690+0000 7fa3c72fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:43.692 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:43.690+0000 7fa3c72fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:44.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:26:44.077 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:26:44.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:44.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:44.250 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:44.250+0000 7fa3c72fe7c0 -1 Falling back to public interface 2024-04-25T08:26:44.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:45.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:45.548 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:26:45.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:45.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:26:45.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:45.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:46.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:47.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:47.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:47.026 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:26:47.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:26:47.026 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:26:47.026+0000 7fa3c72fe7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:26:47.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:47.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:47.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:48.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:48.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:48.610 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:26:48.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:26:48.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:48.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:49.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:50.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:50.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:50.103 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:26:50.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:26:50.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:50.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:50.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:51.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:51.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:51.788 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:26:51.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:26:51.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:51.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:52.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:53.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:53.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:53.306 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:26:53.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:26:53.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:53.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:53.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:54.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:54.746 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:26:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:26:54.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:54.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:55.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:56.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:56.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:56.267 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:26:56.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:26:56.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:56.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:56.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:57.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:57.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:57.820 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:26:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:26:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:57.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:58.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:59.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:59.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:59.310 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:26:59.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:26:59.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:59.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:26:59.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:00.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:00.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:00.761 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:27:00.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:27:00.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:00.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:01.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:02.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:02.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:02.202 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:27:02.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:27:02.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:02.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:02.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:03.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:03.985 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:27:03.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:03.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:27:03.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:03.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:04.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:05.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:05.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:05.813 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:27:05.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:27:05.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:05.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:06.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:07.543 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:27:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:27:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:07.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:08.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:09.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:09.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:09.708 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:27:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:27:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:09.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:10.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:11.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:11.622 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:27:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:27:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:11.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:12.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:13.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:13.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:13.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:27:13.321 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:27:13.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:13.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:27:13.756 INFO:tasks.workunit.client.0.smithi031.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3865217722,v1:127.0.0.1:6843/3865217722] [v2:127.0.0.1:6844/3865217722,v1:127.0.0.1:6845/3865217722] exists,up 2bfcfbe5-30ce-4428-84e0-959648658418 2024-04-25T08:27:13.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:27:13.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:27:13.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:27:13.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-04-25T08:27:13.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:27:14.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-04-25T08:27:14.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:27:15.543 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:27:15.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:27:16.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:27:16.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:27:16.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:27:16.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:27:16.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:27:16.567 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:27:16.568 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:27:16.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:27:16.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:27:16.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:27:16.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:27:16.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:27:16.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:27:16.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:27:16.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:27:16.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:27:17.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:17.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:27:17.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836534 2024-04-25T08:27:17.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836534 2024-04-25T08:27:17.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534' 2024-04-25T08:27:17.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:17.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:27:17.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672999 2024-04-25T08:27:17.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672999 2024-04-25T08:27:17.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949672999' 2024-04-25T08:27:17.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:17.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:27:17.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509470 2024-04-25T08:27:17.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509470 2024-04-25T08:27:17.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949672999 2-64424509470' 2024-04-25T08:27:17.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:17.673 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:27:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T08:27:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T08:27:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949672999 2-64424509470 3-85899345941' 2024-04-25T08:27:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:17.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:27:17.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182413 2024-04-25T08:27:17.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182413 2024-04-25T08:27:17.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949672999 2-64424509470 3-85899345941 4-107374182413' 2024-04-25T08:27:17.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:17.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:27:18.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018885 2024-04-25T08:27:18.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018885 2024-04-25T08:27:18.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949672999 2-64424509470 3-85899345941 4-107374182413 5-128849018885' 2024-04-25T08:27:18.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:18.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836534 2024-04-25T08:27:18.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:18.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:27:18.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836534 2024-04-25T08:27:18.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:18.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836534 2024-04-25T08:27:18.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-04-25T08:27:18.150 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836534 2024-04-25T08:27:18.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:27:18.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836535 -lt 21474836534 2024-04-25T08:27:18.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:18.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672999 2024-04-25T08:27:18.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:18.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:27:18.600 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672999 2024-04-25T08:27:18.600 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:18.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672999 2024-04-25T08:27:18.601 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672999 2024-04-25T08:27:18.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-04-25T08:27:18.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:27:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672999 -lt 42949672999 2024-04-25T08:27:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:19.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509470 2024-04-25T08:27:19.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:27:19.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:19.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509470 2024-04-25T08:27:19.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509470 2024-04-25T08:27:19.028 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509470 2024-04-25T08:27:19.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-04-25T08:27:19.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:27:19.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509470 -lt 64424509470 2024-04-25T08:27:19.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:19.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345941 2024-04-25T08:27:19.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:19.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:27:19.470 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-04-25T08:27:19.470 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:19.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T08:27:19.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-04-25T08:27:19.471 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345941 2024-04-25T08:27:19.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:27:19.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345941 2024-04-25T08:27:19.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:19.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:19.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182413 2024-04-25T08:27:19.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:27:19.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182413 2024-04-25T08:27:19.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:19.917 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182413 2024-04-25T08:27:19.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182413 2024-04-25T08:27:19.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2024-04-25T08:27:19.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:27:20.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182414 -lt 107374182413 2024-04-25T08:27:20.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:20.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018885 2024-04-25T08:27:20.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:20.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:27:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018885 2024-04-25T08:27:20.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:20.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018885 2024-04-25T08:27:20.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2024-04-25T08:27:20.388 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018885 2024-04-25T08:27:20.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:27:20.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018886 -lt 128849018885 2024-04-25T08:27:20.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:27:20.811 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:20.811 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:21.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:27:21.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:21.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:21.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:21.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:21.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:21.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:21.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:21.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:27:21.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:21.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:21.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:22.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:27:22.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:27:22.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:27:22.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:27:22.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:22.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:27:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:27:22.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:22.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:27:23.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:23.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:27:23.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:23.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:27:23.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:23.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:27:23.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:23.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:27:23.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:23.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:27:24.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:24.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:27:24.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:24.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:27:24.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:24.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:27:24.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:24.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:27:24.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:24.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:27:25.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:25.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:27:25.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:25.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:27:25.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:25.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:27:25.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:25.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:27:26.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:26.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:27:26.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:26.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:27:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:26.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:27:26.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:26.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:27:26.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:26.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:27:27.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:27.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:27:27.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:27.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:27:27.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:27.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:27:27.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:27.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:27:27.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:27.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:27:28.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:28.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:27:28.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:28.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:27:28.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:28.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:27:28.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:28.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:27:28.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:28.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:27:29.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:29.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:27:29.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:29.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:27:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:27:29.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:29.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:27:29.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:29.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:27:30.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:30.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:27:30.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:30.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:27:30.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:30.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:27:30.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:30.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:27:31.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:31.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:27:31.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:31.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:27:31.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:31.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:27:31.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:31.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:27:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:31.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:27:32.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:32.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:27:32.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:32.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:27:32.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:32.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:27:32.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:32.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:27:33.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:33.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:27:33.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:33.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:27:33.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:33.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:27:33.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:33.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:27:33.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:27:34.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:34.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:27:34.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:34.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:27:34.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:34.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:27:34.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:34.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:27:34.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:34.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:27:35.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:35.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:27:35.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:35.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:27:35.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:35.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:27:35.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:35.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:27:35.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:35.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:27:36.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:36.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:27:36.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:36.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:27:36.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:36.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:27:36.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:36.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:27:36.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:36.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:27:37.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:37.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:27:37.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:37.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:27:37.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:37.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:27:37.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:37.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:27:37.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:37.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:27:38.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:38.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:27:38.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:38.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:27:38.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:38.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:27:38.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:38.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:27:39.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:39.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:27:39.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:39.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:27:39.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:39.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:27:39.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:39.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:27:39.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:39.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:27:40.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:40.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:27:40.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:40.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:27:40.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:40.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:27:40.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:40.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:27:40.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:40.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:27:41.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:41.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:27:41.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:41.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:27:41.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:41.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:27:41.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:41.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:27:42.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:42.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:27:42.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:42.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:27:42.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:42.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:27:42.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:42.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:27:43.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:43.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:27:43.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:27:43.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:43.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:27:43.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:43.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:27:43.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:43.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:27:44.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:44.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:27:44.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:44.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:27:44.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:44.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:27:44.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:44.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:27:44.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:44.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:27:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:27:45.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:45.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:27:45.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:45.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:27:45.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:45.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:27:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:46.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:27:46.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:46.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:27:46.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:46.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:27:46.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:46.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:27:47.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:47.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:27:47.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:47.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:27:47.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:47.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:27:47.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:47.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:27:47.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:47.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:27:48.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:48.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:27:48.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:48.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:27:48.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:48.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:27:48.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:48.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:27:49.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:49.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:27:49.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:27:49.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:49.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:27:49.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:49.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:27:49.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:49.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:27:50.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:50.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:27:50.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:50.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:27:50.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:50.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:27:50.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:50.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:27:51.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:51.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:27:51.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:51.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:27:51.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:51.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:27:51.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:51.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:27:52.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:52.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:27:52.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:52.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:27:52.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:52.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:27:52.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:52.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:27:52.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:52.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:27:53.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:53.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:27:53.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:53.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:27:53.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:53.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:27:53.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:53.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:27:54.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:54.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:27:54.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:54.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:27:54.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:54.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:27:54.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:54.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:27:54.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:54.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:27:55.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:55.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:27:55.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:55.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:27:55.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:55.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:27:55.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:55.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:27:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:56.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:27:56.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:56.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:27:56.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:56.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:27:56.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:56.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:27:57.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:57.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:27:57.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:57.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:27:57.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:57.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:27:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:27:58.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:58.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:27:58.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:58.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:27:58.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:58.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:27:58.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:58.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:27:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:27:59.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:59.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:27:59.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:59.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:27:59.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:59.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:27:59.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:59.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:27:59.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:27:59.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:28:00.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:00.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:28:00.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:00.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:28:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:28:00.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:00.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:28:01.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:01.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:28:01.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:01.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:28:01.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:01.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:28:01.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:01.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:28:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:02.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:28:02.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:02.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:28:02.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:02.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:28:02.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:02.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:28:02.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:02.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:28:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:03.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:28:03.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:03.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:28:03.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:03.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:28:03.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:03.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:28:04.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:04.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:28:04.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:04.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:28:04.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:04.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:28:04.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:04.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:28:04.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:04.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:28:05.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:05.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:28:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:28:05.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:05.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:28:05.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:05.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:28:05.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:28:05.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:28:06.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:28:06.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:28:06.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:28:06.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:28:06.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:28:06.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:28:06.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:28:06.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:28:06.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:28:06.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:28:06.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:28:07.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:28:07.050 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:28:07.050 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:28:07.050 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:28:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:28:07.051 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:28:07.051 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:28:07.051 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:28:07.051 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:28:07.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:28:07.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:28:07.512 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:28:07.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:28:07.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:28:08.703 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:28:08.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:28:08.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 275668 2024-04-25T08:28:08.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:28:09.242 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T08:28:09.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:28:10.119 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T08:28:10.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:28:11.361 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:28:11.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:28:11.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:28:11.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:28:11.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:28:11.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:28:11.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:28:11.985 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:28:12.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:28:14.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:28:14.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:28:14.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:28:14.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:28:14.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:28:14.004 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:28:14.004 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:28:14.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:28:14.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:28:14.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:28:14.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:28:14.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:28:14.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:28:14.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:28:14.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:28:14.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:28:14.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:28:14.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:28:14.794 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:28:14.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:28:14.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:28:14.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:28:14.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:28:14.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:28:14.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673012 2024-04-25T08:28:14.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673012 2024-04-25T08:28:14.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673012' 2024-04-25T08:28:14.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:28:14.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:28:15.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509483 2024-04-25T08:28:15.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509483 2024-04-25T08:28:15.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673012 2-64424509483' 2024-04-25T08:28:15.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:28:15.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:28:15.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T08:28:15.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T08:28:15.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673012 2-64424509483 3-85899345954' 2024-04-25T08:28:15.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:28:15.271 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:28:15.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182427 2024-04-25T08:28:15.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182427 2024-04-25T08:28:15.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673012 2-64424509483 3-85899345954 4-107374182427' 2024-04-25T08:28:15.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:28:15.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:28:15.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018899 2024-04-25T08:28:15.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018899 2024-04-25T08:28:15.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673012 2-64424509483 3-85899345954 4-107374182427 5-128849018899' 2024-04-25T08:28:15.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:28:15.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673012 2024-04-25T08:28:15.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:28:15.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:28:15.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673012 2024-04-25T08:28:15.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:28:15.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673012 2024-04-25T08:28:15.594 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673012 2024-04-25T08:28:15.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-04-25T08:28:15.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:28:16.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673010 -lt 42949673012 2024-04-25T08:28:16.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:28:17.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:28:17.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:28:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673012 -lt 42949673012 2024-04-25T08:28:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:28:17.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509483 2024-04-25T08:28:17.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:28:17.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:28:17.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509483 2024-04-25T08:28:17.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:28:17.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509483 2024-04-25T08:28:17.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2024-04-25T08:28:17.465 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509483 2024-04-25T08:28:17.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:28:17.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509483 -lt 64424509483 2024-04-25T08:28:17.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:28:17.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:28:17.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345954 2024-04-25T08:28:17.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:28:17.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345954 2024-04-25T08:28:17.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:28:17.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T08:28:17.921 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345954 2024-04-25T08:28:17.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345954' 2024-04-25T08:28:17.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:28:18.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345955 -lt 85899345954 2024-04-25T08:28:18.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:28:18.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182427 2024-04-25T08:28:18.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:28:18.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:28:18.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182427 2024-04-25T08:28:18.374 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:28:18.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182427 2024-04-25T08:28:18.375 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182427 2024-04-25T08:28:18.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-04-25T08:28:18.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:28:18.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182427 -lt 107374182427 2024-04-25T08:28:18.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:28:18.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018899 2024-04-25T08:28:18.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:28:18.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:28:18.882 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018899 2024-04-25T08:28:18.882 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:28:18.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018899 2024-04-25T08:28:18.883 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018899 2024-04-25T08:28:18.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018899' 2024-04-25T08:28:18.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:28:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018899 -lt 128849018899 2024-04-25T08:28:19.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:28:19.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:19.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:28:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:19.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:19.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:19.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:19.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:19.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:19.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:20.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:20.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:20.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:20.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:21.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:21.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:28:21.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:28:21.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:28:21.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:21.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:21.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:21.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:21.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:21.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:21.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:21.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:21.196 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:21.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:21.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:21.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:21.620 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:21.620 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:22.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:22.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:22.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-04-25T08:28:22.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-04-25T08:28:22.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:22.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:22.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:22.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:22.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:22.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:22.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:22.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:22.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:23.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:23.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:23.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:23.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:24.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:24.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:24.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:24.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:24.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:24.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:24.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:24.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:24.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:28:24.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:28:24.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:24.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:24.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:24.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:24.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:24.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:24.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:24.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:24.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:24.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:24.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:25.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:25.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:25.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:25.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:25.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:25.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:26.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:28:26.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:28:26.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:26.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:26.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:26.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:26.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:26.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:26.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:26.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:26.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:26.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:26.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:27.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:27.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:27.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:27.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:27.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:27.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:27.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16 2024-04-25T08:28:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16 '!=' null 2024-04-25T08:28:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:28.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:28.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:28.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:28.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:28.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:28.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:28.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:28.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:29.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:29.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:29.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:29.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:29.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:29.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:29.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:29.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:29.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:29.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:29.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:29.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:29.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:30.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16 2024-04-25T08:28:30.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16 '!=' null 2024-04-25T08:28:30.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:30.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:30.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:30.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:30.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:30.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:30.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:30.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:30.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:30.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:31.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:31.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:31.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:28:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:28:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:28:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:31.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:31.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:31.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:31.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:31.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:31.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:31.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:31.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:32.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:28:32.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:32.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:32.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:28:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:28:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:28:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-04-25T08:28:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:28:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:28:32.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:28:32.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:28:32.881 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:28:32.881 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:28:32.881 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:28:32.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-04-25T08:28:32.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:28:32.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:28:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:28:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:28:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:28:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:28:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:28:32.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:28:32.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:28:32.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:28:32.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:28:32.918 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:28:32.918 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:28:32.918 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:28:32.918 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:28:32.918 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:28:32.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:28:32.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:28:32.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:28:32.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:28:32.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:28:32.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:28:32.927 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:28:32.927 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:28:32.927 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:28:32.927 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:28:32.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:28:32.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:28:33.450 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:28:33.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:28:33.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:28:33.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:28:33.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:28:33.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:28:33.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:28:38.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:28:38.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:28:38.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:28:38.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:28:38.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:28:38.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:28:38.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:28:38.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:28:38.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:28:38.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:28:38.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:28:38.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:28:38.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:28:38.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:28:38.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:28:38.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:28:38.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:28:38.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:28:38.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:38.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:38.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:28:38.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:28:38.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:28:38.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:28:38.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:28:38.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:28:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:28:38.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:28:38.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:28:38.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:28:38.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:28:38.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:28:38.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:28:38.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:28:38.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:28:38.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:28:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:28:38.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:28:38.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:28:38.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:28:38.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:28:38.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:28:38.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:38.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:38.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:28:38.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:28:38.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:28:38.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:28:38.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:28:38.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:38.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:38.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:28:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:28:38.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:28:38.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:28:38.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:28:38.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:28:39.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:28:39.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:28:39.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:28:39.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:28:39.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:39.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:39.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:28:39.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:28:39.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:28:39.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:28:39.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:28:39.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:28:39.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:28:39.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:28:39.596 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:28:39.596 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:28:39.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:28:39.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:28:39.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:28:39.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:28:39.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:28:39.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:28:39.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:28:39.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:28:39.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:28:39.719 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:28:39.719 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:39.719 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:39.719 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:28:39.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:28:39.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:28:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:28:40.772 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:28:40.772 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:28:40.772 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:28:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:28:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:40.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:28:40.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:28:40.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:28:40.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:28:40.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:28:40.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:28:40.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:28:40.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:28:40.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:28:40.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af12336b-d264-43fd-bdc9-93bd3220d8fd 2024-04-25T08:28:40.849 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 af12336b-d264-43fd-bdc9-93bd3220d8fd 2024-04-25T08:28:40.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 af12336b-d264-43fd-bdc9-93bd3220d8fd' 2024-04-25T08:28:40.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:28:40.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4FCpmO6QFNBAA7NaGOXztmPEmk1APr0Budg== 2024-04-25T08:28:40.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4FCpmO6QFNBAA7NaGOXztmPEmk1APr0Budg=="}' 2024-04-25T08:28:40.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af12336b-d264-43fd-bdc9-93bd3220d8fd -i td/osd-recovery-stats/0/new.json 2024-04-25T08:28:41.199 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:28:41.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:28:41.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4FCpmO6QFNBAA7NaGOXztmPEmk1APr0Budg== --osd-uuid af12336b-d264-43fd-bdc9-93bd3220d8fd 2024-04-25T08:28:41.254 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.256+0000 7fed988777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:41.276 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.280+0000 7fed988777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:41.278 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.280+0000 7fed988777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:41.279 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.280+0000 7fed988777c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:28:41.279 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.280+0000 7fed988777c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:28:41.279 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.280+0000 7fed988777c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:28:41.316 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:41.320+0000 7fed988777c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:28:47.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:28:47.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:28:47.980 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:28:47.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:28:47.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:28:48.553 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:28:48.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:28:48.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:28:48.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:28:48.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:28:48.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:28:48.587 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:48.588+0000 7f40f80b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:48.597 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:48.600+0000 7f40f80b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:48.598 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:48.600+0000 7f40f80b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:48.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:28:48.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:28:48.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:48.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:49.141 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:49.144+0000 7f40f80b07c0 -1 Falling back to public interface 2024-04-25T08:28:49.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:50.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:50.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:50.417 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:28:50.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:28:50.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:50.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:50.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:51.854 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:28:51.856+0000 7f40f80b07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:28:51.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:51.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:51.856 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:28:51.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:28:51.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:51.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:52.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:53.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:53.407 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:28:53.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:28:53.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:53.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:53.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:54.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:54.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:54.904 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:28:54.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:28:54.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:54.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:55.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:56.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:56.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:56.382 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:28:56.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:28:56.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:56.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:56.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:57.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:57.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:57.921 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:28:57.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:28:57.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:57.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:58.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:59.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:59.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:59.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:28:59.395 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:28:59.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:59.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:00.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:00.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:00.933 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:29:00.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:29:00.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:00.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:02.388 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:29:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:29:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:02.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:02.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:03.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:03.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:03.927 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:29:03.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:29:03.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:03.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:04.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:05.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:05.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:05.405 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:29:05.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:29:05.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:05.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:05.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:06.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:06.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:06.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:29:06.937 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:29:06.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:06.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:07.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:08.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:08.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:08.993 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:29:08.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:29:08.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:08.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:10.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:11.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:11.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:11.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:29:11.164 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:29:11.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:11.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:12.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:13.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:13.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:13.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:29:13.030 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:29:13.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:13.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:13.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:14.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:14.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:14.855 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:29:14.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:29:14.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:14.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:15.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:16.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:16.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:16.754 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:29:16.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:29:16.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:16.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:17.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:18.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:18.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:18.355 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:29:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:29:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:29:18.781 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2380096349,v1:127.0.0.1:6803/2380096349] [v2:127.0.0.1:6804/2380096349,v1:127.0.0.1:6805/2380096349] exists,up af12336b-d264-43fd-bdc9-93bd3220d8fd 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:29:18.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:29:18.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:29:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:29:18.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=547ef388-93ab-4eb1-8824-d2c73ecc6050 2024-04-25T08:29:18.789 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 547ef388-93ab-4eb1-8824-d2c73ecc6050 2024-04-25T08:29:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 547ef388-93ab-4eb1-8824-d2c73ecc6050' 2024-04-25T08:29:18.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:29:18.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeFCpmCs2KMBAAtvgdrV54vreuU1yYEM+P+A== 2024-04-25T08:29:18.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeFCpmCs2KMBAAtvgdrV54vreuU1yYEM+P+A=="}' 2024-04-25T08:29:18.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 547ef388-93ab-4eb1-8824-d2c73ecc6050 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:29:19.329 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:29:19.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:29:19.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeFCpmCs2KMBAAtvgdrV54vreuU1yYEM+P+A== --osd-uuid 547ef388-93ab-4eb1-8824-d2c73ecc6050 2024-04-25T08:29:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.383+0000 7f4760d367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:19.417 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.419+0000 7f4760d367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:19.419 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.423+0000 7f4760d367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:19.420 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.423+0000 7f4760d367c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:29:19.420 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.423+0000 7f4760d367c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:29:19.420 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.423+0000 7f4760d367c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:29:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:19.451+0000 7f4760d367c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:29:26.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:29:26.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:29:26.131 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:29:26.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:29:26.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:29:26.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:29:26.699 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:29:26.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:29:26.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:29:26.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:29:26.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:29:26.725 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:26.727+0000 7fa99bd3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:26.757 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:26.759+0000 7fa99bd3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:26.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:26.759+0000 7fa99bd3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:27.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:29:27.139 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:29:27.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:27.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:27.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:27.327+0000 7fa99bd3e7c0 -1 Falling back to public interface 2024-04-25T08:29:27.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:28.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:28.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:28.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:29:28.583 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:29:28.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:28.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:29.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:30.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:30.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:30.013 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:29:30.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:29:30.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:30.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:30.160 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:30.163+0000 7fa99bd3e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:29:30.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:31.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:31.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:31.515 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:29:31.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:29:31.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:31.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:32.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:33.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:33.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:33.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:29:33.104 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:29:33.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:33.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:33.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:34.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:34.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:34.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:29:34.552 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:29:34.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:34.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:35.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:36.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:36.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:36.076 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:29:36.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:29:36.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:36.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:36.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:37.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:37.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:37.675 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:29:37.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:29:37.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:37.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:38.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:39.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:39.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:39.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:29:39.240 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:29:39.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:39.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:39.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:40.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:40.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:40.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:29:40.712 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:29:40.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:40.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:42.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:42.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:42.198 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:29:42.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:29:42.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:42.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:43.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:43.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:43.632 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:29:43.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:29:43.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:43.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:44.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:45.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:45.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:45.068 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:29:45.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:29:45.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:45.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:45.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:46.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:46.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:46.890 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:29:46.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:29:46.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:46.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:47.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:48.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:48.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:48.601 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:29:48.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:29:48.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:48.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:49.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:50.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:50.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:50.356 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:29:50.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:29:50.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:50.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:51.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:52.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:52.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:52.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:29:52.230 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:29:52.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:52.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:53.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:54.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:54.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:54.127 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:29:54.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:29:54.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:54.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:54.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:55.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:55.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:55.959 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:29:55.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:29:55.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:55.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:56.083 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:56.086+0000 7fa997cf8640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T08:29:56.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:57.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:57.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:57.462 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T08:29:57.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:29:57.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:57.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:57.923 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4265172554,v1:127.0.0.1:6811/4265172554] [v2:127.0.0.1:6812/4265172554,v1:127.0.0.1:6813/4265172554] exists,up 547ef388-93ab-4eb1-8824-d2c73ecc6050 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:29:57.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:29:57.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:29:57.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:29:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:29:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:29:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:29:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:29:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:29:57.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:29:57.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3cc78d0e-3a16-4862-8c80-8e2bfca15a3f 2024-04-25T08:29:57.931 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 3cc78d0e-3a16-4862-8c80-8e2bfca15a3f 2024-04-25T08:29:57.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3cc78d0e-3a16-4862-8c80-8e2bfca15a3f' 2024-04-25T08:29:57.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:29:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFFCpmMfYYORAA78+ZU/fjBapybZiVt5Q4KQ== 2024-04-25T08:29:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFFCpmMfYYORAA78+ZU/fjBapybZiVt5Q4KQ=="}' 2024-04-25T08:29:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3cc78d0e-3a16-4862-8c80-8e2bfca15a3f -i td/osd-recovery-stats/2/new.json 2024-04-25T08:29:58.460 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:29:58.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:29:58.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFFCpmMfYYORAA78+ZU/fjBapybZiVt5Q4KQ== --osd-uuid 3cc78d0e-3a16-4862-8c80-8e2bfca15a3f 2024-04-25T08:29:58.516 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.518+0000 7fa8fea857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:58.543 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.546+0000 7fa8fea857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:58.544 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.546+0000 7fa8fea857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:58.544 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.546+0000 7fa8fea857c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:29:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.546+0000 7fa8fea857c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:29:58.545 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.546+0000 7fa8fea857c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:29:58.598 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:29:58.598+0000 7fa8fea857c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:30:05.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:30:05.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:30:05.597 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:30:05.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:30:05.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:30:06.200 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:30:06.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:30:06.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:30:06.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:30:06.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:30:06.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:30:06.234 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:06.234+0000 7f38e19b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:06.236 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:06.238+0000 7f38e19b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:06.238 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:06.238+0000 7f38e19b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:06.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:30:06.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:30:06.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:06.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:06.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:06.826+0000 7f38e19b97c0 -1 Falling back to public interface 2024-04-25T08:30:07.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:08.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:08.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:08.114 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:30:08.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:30:08.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:08.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:08.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:09.516 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:09.518+0000 7f38e19b97c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:30:09.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:09.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:09.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:30:09.542 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:30:09.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:09.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:10.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:11.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:11.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:11.137 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:30:11.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:30:11.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:11.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:11.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:12.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:12.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:12.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:30:12.590 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:30:12.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:12.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:13.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:14.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:14.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:14.049 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:30:14.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:30:14.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:14.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:14.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:15.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:15.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:15.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:30:15.659 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:30:15.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:15.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:16.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:17.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:17.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:17.316 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:30:17.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:30:17.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:17.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:17.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:18.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:18.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:18.773 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:30:18.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:30:18.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:18.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:19.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:20.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:20.264 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:30:20.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:30:20.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:20.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:20.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:21.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:21.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:21.722 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:30:21.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:30:21.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:21.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:22.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:23.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:23.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:23.311 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:30:23.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:30:23.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:23.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:23.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:24.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:24.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:24.986 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:30:24.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:30:24.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:24.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:25.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:26.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:26.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:26.603 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:30:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:30:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:27.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:28.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:28.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:28.196 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:30:28.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:30:28.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:28.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:28.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:29.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:29.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:29.850 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:30:29.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:30:29.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:29.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:30.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:31.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:31.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:31.797 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:30:31.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:30:31.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:31.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:32.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:33.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:33.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:33.290 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:30:33.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:30:33.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:33.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:34.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:35.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:35.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:35.361 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:30:35.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:30:35.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:35.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:35.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:36.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:36.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:36.877 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T08:30:36.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:30:36.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:36.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:30:37.319 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1403240310,v1:127.0.0.1:6819/1403240310] [v2:127.0.0.1:6820/1403240310,v1:127.0.0.1:6821/1403240310] exists,up 3cc78d0e-3a16-4862-8c80-8e2bfca15a3f 2024-04-25T08:30:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:30:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:30:37.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:30:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:30:37.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:30:37.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3eadf69-38fa-43df-bf06-a41da5066355 2024-04-25T08:30:37.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b3eadf69-38fa-43df-bf06-a41da5066355' 2024-04-25T08:30:37.327 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 b3eadf69-38fa-43df-bf06-a41da5066355 2024-04-25T08:30:37.327 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:30:37.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtFCpmI0f4FBAAenc7Ph0ufWFKO84f4boiwQ== 2024-04-25T08:30:37.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtFCpmI0f4FBAAenc7Ph0ufWFKO84f4boiwQ=="}' 2024-04-25T08:30:37.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3eadf69-38fa-43df-bf06-a41da5066355 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:30:37.850 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:30:37.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:30:37.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtFCpmI0f4FBAAenc7Ph0ufWFKO84f4boiwQ== --osd-uuid b3eadf69-38fa-43df-bf06-a41da5066355 2024-04-25T08:30:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:37.906+0000 7f774252d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:37.949 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:37.950+0000 7f774252d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:37.950 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:37.954+0000 7f774252d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:37.951 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:37.954+0000 7f774252d7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:30:37.951 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:37.954+0000 7f774252d7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:30:37.951 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:37.954+0000 7f774252d7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:30:38.039 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:38.042+0000 7f774252d7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:30:44.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:30:44.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:30:44.886 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:30:44.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:30:44.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:30:45.526 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:30:45.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:30:45.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:30:45.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:30:45.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:30:45.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:30:45.559 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:45.562+0000 7f77522f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:45.580 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:45.582+0000 7f77522f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:45.582 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:45.582+0000 7f77522f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:46.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:30:46.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:30:46.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:30:46.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:30:46.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:30:46.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:46.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:30:46.011 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:30:46.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:46.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:46.159 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:46.162+0000 7f77522f97c0 -1 Falling back to public interface 2024-04-25T08:30:46.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:47.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:47.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:47.474 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:30:47.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:30:47.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:47.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:47.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:48.905 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:30:48.906+0000 7f77522f97c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:30:48.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:48.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:48.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:30:48.910 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:30:48.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:48.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:49.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:50.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:50.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:50.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:30:50.516 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:30:50.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:50.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:51.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:52.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:52.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:52.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:30:52.027 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:30:52.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:52.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:52.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:53.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:53.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:53.649 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:30:53.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:30:53.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:53.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:54.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:55.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:55.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:55.308 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:30:55.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:30:55.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:55.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:55.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:56.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:56.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:56.916 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:30:56.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:30:56.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:56.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:57.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:58.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:58.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:58.391 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:30:58.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:30:58.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:58.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:58.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:59.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:59.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:59.880 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:30:59.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:30:59.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:59.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:00.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:01.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:01.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:01.379 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:31:01.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:31:01.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:01.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:01.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:02.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:02.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:02.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:31:02.915 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:31:02.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:02.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:03.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:04.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:04.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:04.425 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:31:04.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:31:04.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:04.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:05.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:06.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:06.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:06.401 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:31:06.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:31:06.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:06.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:07.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:08.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:08.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:08.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:31:08.495 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:31:08.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:08.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:09.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:10.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:10.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:10.312 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:31:10.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:31:10.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:10.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:11.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:12.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:12.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:12.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:31:12.139 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:31:12.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:12.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:13.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:14.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:14.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:14.241 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:31:14.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:31:14.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:14.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:14.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:15.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:15.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:15.727 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:31:15.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:31:15.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:15.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:31:16.206 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/520004904,v1:127.0.0.1:6827/520004904] [v2:127.0.0.1:6828/520004904,v1:127.0.0.1:6829/520004904] exists,up b3eadf69-38fa-43df-bf06-a41da5066355 2024-04-25T08:31:16.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:31:16.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:31:16.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:31:16.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:31:16.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-04-25T08:31:16.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:31:17.850 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:31:17.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:31:18.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-04-25T08:31:19.729 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 2 2024-04-25T08:31:19.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:31:19.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:31:19.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:31:19.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:31:19.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:31:19.754 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:31:19.754 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:31:19.754 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:31:19.755 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:31:19.755 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:31:19.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:31:19.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:31:19.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:31:19.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:31:19.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:31:19.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:31:20.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:31:20.612 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:31:20.612 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:31:20.612 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:31:20.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:31:20.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:20.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:31:20.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-04-25T08:31:20.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-04-25T08:31:20.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-04-25T08:31:20.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:20.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:31:20.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T08:31:20.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T08:31:20.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672983' 2024-04-25T08:31:20.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:20.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:31:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-04-25T08:31:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-04-25T08:31:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672983 2-64424509457' 2024-04-25T08:31:21.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:21.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:31:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345928 2024-04-25T08:31:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345928 2024-04-25T08:31:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672983 2-64424509457 3-85899345928' 2024-04-25T08:31:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:21.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-04-25T08:31:21.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:21.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:31:21.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-04-25T08:31:21.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:21.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-04-25T08:31:21.254 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836513 2024-04-25T08:31:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-04-25T08:31:21.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:31:21.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836513 2024-04-25T08:31:21.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:31:22.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:31:22.673 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:31:23.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-04-25T08:31:23.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:23.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T08:31:23.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:23.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:31:23.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T08:31:23.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:23.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T08:31:23.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T08:31:23.100 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672983 2024-04-25T08:31:23.100 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:31:23.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T08:31:23.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:23.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:23.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-04-25T08:31:23.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:31:23.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:23.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-04-25T08:31:23.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-04-25T08:31:23.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-25T08:31:23.590 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509457 2024-04-25T08:31:23.590 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:31:24.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509457 2024-04-25T08:31:24.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:24.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:24.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345928 2024-04-25T08:31:24.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:31:24.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345928 2024-04-25T08:31:24.074 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:24.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345928 2024-04-25T08:31:24.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-04-25T08:31:24.078 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345928 2024-04-25T08:31:24.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:31:24.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345928 -lt 85899345928 2024-04-25T08:31:24.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:31:24.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:24.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:25.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:25.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:25.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:25.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:25.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:25.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:26.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:31:26.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:31:26.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:31:26.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-04-25T08:31:26.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:26.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-04-25T08:31:26.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:26.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-04-25T08:31:26.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:26.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-04-25T08:31:26.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:26.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-04-25T08:31:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:26.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-04-25T08:31:27.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:27.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-04-25T08:31:27.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:27.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-04-25T08:31:27.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:27.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-04-25T08:31:27.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:27.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-04-25T08:31:28.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:28.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-04-25T08:31:28.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:28.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-04-25T08:31:28.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:28.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-04-25T08:31:28.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:28.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-04-25T08:31:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-04-25T08:31:29.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:29.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-04-25T08:31:29.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:29.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-04-25T08:31:29.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:29.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-04-25T08:31:29.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:29.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-04-25T08:31:30.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:30.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-04-25T08:31:30.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:30.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-04-25T08:31:30.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:30.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-04-25T08:31:30.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:30.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-04-25T08:31:30.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:30.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-04-25T08:31:31.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:31.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-04-25T08:31:31.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:31.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-04-25T08:31:31.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:31.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-04-25T08:31:31.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:31.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-04-25T08:31:32.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:32.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-04-25T08:31:32.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:32.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-04-25T08:31:32.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:32.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-04-25T08:31:32.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:32.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-04-25T08:31:33.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:33.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-04-25T08:31:33.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:33.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-04-25T08:31:33.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:33.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-04-25T08:31:33.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:33.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-04-25T08:31:34.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:34.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-04-25T08:31:34.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:34.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-04-25T08:31:34.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:34.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-04-25T08:31:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-04-25T08:31:34.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:34.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-04-25T08:31:35.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:35.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-04-25T08:31:35.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-04-25T08:31:35.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:35.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-04-25T08:31:35.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:35.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-04-25T08:31:36.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:36.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-04-25T08:31:36.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:36.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-04-25T08:31:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-04-25T08:31:36.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:36.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-04-25T08:31:37.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:37.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-04-25T08:31:37.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:37.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-04-25T08:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-04-25T08:31:37.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:37.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-04-25T08:31:37.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:37.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-04-25T08:31:38.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:38.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-04-25T08:31:38.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:38.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-04-25T08:31:38.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:38.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-04-25T08:31:38.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:38.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-04-25T08:31:39.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:39.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-04-25T08:31:39.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:39.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-04-25T08:31:39.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:39.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-04-25T08:31:39.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:39.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-04-25T08:31:39.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:39.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-04-25T08:31:40.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:40.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-04-25T08:31:40.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:40.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-04-25T08:31:40.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:40.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-04-25T08:31:40.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:40.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-04-25T08:31:41.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:41.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-04-25T08:31:41.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:41.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-04-25T08:31:41.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:41.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-04-25T08:31:41.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:41.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-04-25T08:31:42.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:42.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-04-25T08:31:42.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:42.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-04-25T08:31:42.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:42.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-04-25T08:31:42.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:42.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-04-25T08:31:43.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:43.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-04-25T08:31:43.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:43.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-04-25T08:31:43.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:43.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-04-25T08:31:43.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:43.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-04-25T08:31:44.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:44.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-04-25T08:31:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-04-25T08:31:44.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:44.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-04-25T08:31:45.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:45.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-04-25T08:31:45.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:45.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-04-25T08:31:45.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:45.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-04-25T08:31:45.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:45.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-04-25T08:31:45.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:45.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-04-25T08:31:46.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:46.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-04-25T08:31:46.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:46.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-04-25T08:31:46.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:46.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-04-25T08:31:47.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:47.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-04-25T08:31:47.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:47.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-04-25T08:31:47.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:47.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-04-25T08:31:47.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:47.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-04-25T08:31:48.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:48.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-04-25T08:31:48.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:48.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-04-25T08:31:48.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:48.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-04-25T08:31:48.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:48.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-04-25T08:31:49.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:49.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-04-25T08:31:49.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:49.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-04-25T08:31:49.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:49.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-04-25T08:31:49.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:49.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-04-25T08:31:49.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:49.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-04-25T08:31:50.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:50.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-04-25T08:31:50.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:50.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-04-25T08:31:50.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:50.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-04-25T08:31:50.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:50.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-04-25T08:31:51.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:51.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-04-25T08:31:51.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:51.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-04-25T08:31:51.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:51.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-04-25T08:31:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-04-25T08:31:52.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:52.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-04-25T08:31:52.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:52.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-04-25T08:31:52.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:52.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-04-25T08:31:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-04-25T08:31:53.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:53.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-04-25T08:31:53.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:53.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-04-25T08:31:53.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:53.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-04-25T08:31:53.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:53.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-04-25T08:31:53.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:53.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-04-25T08:31:54.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:54.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-04-25T08:31:54.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:54.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-04-25T08:31:54.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:54.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-04-25T08:31:54.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:54.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-04-25T08:31:55.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:55.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-04-25T08:31:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-04-25T08:31:55.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:55.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-04-25T08:31:55.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:55.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-04-25T08:31:56.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:56.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-04-25T08:31:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-04-25T08:31:56.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:56.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-04-25T08:31:56.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:56.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-04-25T08:31:56.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:56.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-04-25T08:31:57.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:57.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-04-25T08:31:57.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:57.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-04-25T08:31:57.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:57.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-04-25T08:31:57.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:57.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-04-25T08:31:58.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:58.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-04-25T08:31:58.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:58.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-04-25T08:31:58.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:58.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-04-25T08:31:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-04-25T08:31:59.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:59.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-04-25T08:31:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:59.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-04-25T08:31:59.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:59.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-04-25T08:31:59.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:31:59.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-04-25T08:32:00.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:00.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-04-25T08:32:00.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:00.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-04-25T08:32:00.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:00.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-04-25T08:32:00.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:00.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-04-25T08:32:00.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:00.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-04-25T08:32:01.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:01.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-04-25T08:32:01.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:01.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-04-25T08:32:01.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:01.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-04-25T08:32:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-04-25T08:32:02.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:02.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-04-25T08:32:02.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:02.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-04-25T08:32:02.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:02.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-04-25T08:32:02.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:02.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-04-25T08:32:03.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:03.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-04-25T08:32:03.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:03.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-04-25T08:32:03.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:03.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-04-25T08:32:03.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:03.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-04-25T08:32:03.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:03.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-04-25T08:32:04.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:04.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-04-25T08:32:04.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:04.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-04-25T08:32:04.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:04.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-04-25T08:32:04.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:04.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-04-25T08:32:05.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:05.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-04-25T08:32:05.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:05.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-04-25T08:32:05.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:05.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-04-25T08:32:05.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:05.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-04-25T08:32:06.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:06.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-04-25T08:32:06.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:06.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-04-25T08:32:06.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:06.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-04-25T08:32:06.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:06.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-04-25T08:32:06.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:06.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-04-25T08:32:07.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:07.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-04-25T08:32:07.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:07.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-04-25T08:32:07.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:07.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-04-25T08:32:07.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:07.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-04-25T08:32:08.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:08.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-04-25T08:32:08.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:08.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-04-25T08:32:08.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:08.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-04-25T08:32:08.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:08.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-04-25T08:32:09.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:09.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-04-25T08:32:09.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:09.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-04-25T08:32:09.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:09.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-04-25T08:32:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-04-25T08:32:09.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:09.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-04-25T08:32:10.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:10.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-04-25T08:32:10.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:10.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-04-25T08:32:10.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:10.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-04-25T08:32:10.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:10.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-04-25T08:32:11.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:11.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-04-25T08:32:11.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:11.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-04-25T08:32:11.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:11.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-04-25T08:32:11.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:11.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-04-25T08:32:12.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:12.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-04-25T08:32:12.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:12.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-04-25T08:32:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:32:12.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-04-25T08:32:12.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:32:12.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:32:12.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:32:12.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:32:12.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:32:13.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-04-25T08:32:13.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-04-25T08:32:13.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:32:13.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:32:13.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:32:13.276 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:32:13.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-04-25T08:32:13.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-04-25T08:32:13.711 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:32:13.711 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:32:13.711 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:32:13.711 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:32:13.711 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:32:13.711 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:32:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:32:14.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:32:14.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:32:14.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:32:14.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-04-25T08:32:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-04-25T08:32:15.315 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:32:15.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:32:15.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 292360 2024-04-25T08:32:15.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-04-25T08:32:15.902 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T08:32:15.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-04-25T08:32:16.730 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T08:32:16.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:32:16.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-04-25T08:32:16.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-04-25T08:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already out. 2024-04-25T08:32:18.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-04-25T08:32:18.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-04-25T08:32:19.550 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:32:19.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:32:19.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:32:19.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:32:19.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:32:19.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:32:20.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:32:20.460 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:32:20.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-04-25T08:32:22.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:32:22.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:32:22.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:32:22.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:32:22.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:32:22.481 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:32:22.481 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:32:22.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:32:22.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:32:22.482 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:32:22.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:32:22.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:32:22.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:32:22.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:32:22.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:32:22.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:23.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:32:23.218 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:32:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:32:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:32:23.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:32:23.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:23.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:32:23.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-04-25T08:32:23.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-04-25T08:32:23.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672998' 2024-04-25T08:32:23.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:23.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:32:23.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509472 2024-04-25T08:32:23.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509472 2024-04-25T08:32:23.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672998 2-64424509472' 2024-04-25T08:32:23.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:23.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:32:23.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T08:32:23.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T08:32:23.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672998 2-64424509472 3-85899345942' 2024-04-25T08:32:23.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:23.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-04-25T08:32:23.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:23.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:32:23.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-04-25T08:32:23.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:23.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-04-25T08:32:23.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-04-25T08:32:23.741 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672998 2024-04-25T08:32:23.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:32:24.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-04-25T08:32:24.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:24.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509472 2024-04-25T08:32:24.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:24.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:32:24.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509472 2024-04-25T08:32:24.267 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:24.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509472 2024-04-25T08:32:24.268 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509472 2024-04-25T08:32:24.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-04-25T08:32:24.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:32:24.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509472 -lt 64424509472 2024-04-25T08:32:24.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:24.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345942 2024-04-25T08:32:24.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:24.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:32:24.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345942 2024-04-25T08:32:24.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:24.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T08:32:24.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-04-25T08:32:24.797 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345942 2024-04-25T08:32:24.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:32:25.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345940 -lt 85899345942 2024-04-25T08:32:25.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:32:26.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:32:26.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:32:26.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345943 -lt 85899345942 2024-04-25T08:32:26.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:32:26.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:26.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:27.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:32:27.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:27.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:27.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:27.396 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:27.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:27.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:27.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:27.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:27.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:27.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:27.836 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:28.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:28.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:32:28.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:32:28.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:32:28.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:28.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:28.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:28.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:28.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:28.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:28.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:28.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:28.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:28.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:28.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:28.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:28.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:28.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:29.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:29.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:30.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:32:30.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:32:30.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:30.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:30.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:30.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:30.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:30.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:30.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:30.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:30.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:30.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:30.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:30.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:30.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:30.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:30.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:31.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:31.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:32:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:32:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:32.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:32.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:32.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:32.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:32.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:32.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:32.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:32.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:32.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:32.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:32.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:32.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:33.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:33.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:33.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:33.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:33.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:33.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:33.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:33.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:33.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:33.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:34.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:34.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:34.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:34.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:34.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:34.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:32:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:32:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:35.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:35.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:35.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:35.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:35.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:35.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:35.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:35.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:35.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:36.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:36.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:36.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:36.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:36.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:36.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:36.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:37.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:32:37.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:32:37.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:37.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:37.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:37.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:37.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:37.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:37.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:37.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:37.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:37.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:37.380 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:37.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:32:37.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:37.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:37.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:38.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:32:38.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:32:38.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:32:38.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.351 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:32:38.352 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:32:38.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-04-25T08:32:38.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:32:38.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:32:38.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:32:38.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:32:38.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:32:38.374 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:32:38.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:32:38.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:32:38.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:32:38.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:32:38.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:32:38.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:32:38.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:32:38.381 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.381 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:32:38.381 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:32:38.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:32:38.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:32:38.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:32:38.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:32:38.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:32:38.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:32:38.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:32:38.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:32:38.393 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.393 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:32:38.394 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:32:38.394 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:32:38.394 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:32:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-04-25T08:32:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-04-25T08:32:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:32:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:32:38.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:32:38.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:32:38.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:32:38.400 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:32:38.400 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:32:38.400 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:32:38.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-04-25T08:32:38.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-04-25T08:32:38.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:32:38.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:32:38.944 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:32:38.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-04-25T08:32:38.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:32:38.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:32:38.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:32:38.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:32:38.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:32:44.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:32:44.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:32:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:32:44.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:32:44.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:32:44.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:32:44.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:32:44.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:32:44.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:32:44.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:32:44.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:32:44.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:32:44.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:32:44.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:32:44.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:32:44.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:32:44.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:32:44.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:32:44.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:32:44.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:44.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:44.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:32:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:32:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:32:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:32:44.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:32:44.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:32:44.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:32:44.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:32:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:32:44.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:32:44.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:32:44.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:32:44.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:32:44.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:32:44.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:32:44.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:32:44.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:32:44.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:32:44.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:32:44.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:44.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:44.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:32:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:32:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:32:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:32:44.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:32:44.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:44.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:44.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:32:44.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:32:44.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:32:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:32:44.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:32:44.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:32:44.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:32:44.840 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:32:44.840 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:44.840 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:44.840 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:32:44.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:32:45.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:32:45.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:32:45.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:32:45.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:32:45.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:32:45.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:32:45.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:32:45.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:32:45.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:32:45.068 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:32:45.068 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:45.068 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:45.068 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:32:45.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:32:45.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:32:45.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:32:45.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:32:45.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:32:45.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:32:45.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:32:45.186 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:32:45.186 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:32:45.186 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:32:45.187 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:32:45.187 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:45.187 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:45.187 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:32:45.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:32:45.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:32:45.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:32:46.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:32:46.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:32:46.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:32:46.262 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:32:46.262 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:46.262 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:46.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:32:46.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:32:46.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:32:46.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-04-25T08:32:46.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-04-25T08:32:46.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:32:46.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-04-25T08:32:46.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:32:46.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:32:46.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:32:46.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:32:46.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:32:46.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=baf8edda-a00e-4bb1-be52-579a9eab7ed7 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 baf8edda-a00e-4bb1-be52-579a9eab7ed7' 2024-04-25T08:32:46.299 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 baf8edda-a00e-4bb1-be52-579a9eab7ed7 2024-04-25T08:32:46.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:32:46.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuFSpmGx0UExAABD/vEM7q1/dvQNRW4c1sVw== 2024-04-25T08:32:46.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuFSpmGx0UExAABD/vEM7q1/dvQNRW4c1sVw=="}' 2024-04-25T08:32:46.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new baf8edda-a00e-4bb1-be52-579a9eab7ed7 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:32:46.636 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:32:46.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:32:46.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuFSpmGx0UExAABD/vEM7q1/dvQNRW4c1sVw== --osd-uuid baf8edda-a00e-4bb1-be52-579a9eab7ed7 2024-04-25T08:32:46.681 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.687+0000 7f8868d197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:46.714 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.719+0000 7f8868d197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:46.715 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.719+0000 7f8868d197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:46.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.719+0000 7f8868d197c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:32:46.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.719+0000 7f8868d197c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:32:46.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.719+0000 7f8868d197c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:32:46.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:46.755+0000 7f8868d197c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:32:53.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:32:53.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:32:53.321 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:32:53.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:32:53.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:32:53.898 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:32:53.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:32:53.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:32:53.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:32:53.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:32:53.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:32:53.933 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:53.935+0000 7fe03ab9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:53.950 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:53.955+0000 7fe03ab9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:53.951 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:53.955+0000 7fe03ab9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:32:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:32:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:32:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:32:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:32:54.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:54.342 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:32:54.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:32:54.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:54.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:54.538 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:54.543+0000 7fe03ab9b7c0 -1 Falling back to public interface 2024-04-25T08:32:54.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:55.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:55.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:55.779 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:32:55.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:32:55.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:55.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:56.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:57.165 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:32:57.167+0000 7fe03ab9b7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:32:57.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:57.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:57.224 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:32:57.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:32:57.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:57.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:57.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:58.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:58.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:58.933 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:32:58.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:32:58.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:58.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:59.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:00.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:00.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:00.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:33:00.399 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:33:00.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:00.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:00.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:01.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:01.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:01.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:33:01.847 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:33:01.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:01.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:02.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:03.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:03.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:03.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:33:03.350 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:33:03.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:03.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:03.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:04.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:04.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:04.793 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:33:04.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:33:04.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:04.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:05.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:06.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:06.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:06.283 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:33:06.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:33:06.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:06.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:06.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:07.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:07.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:07.728 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:33:07.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:33:07.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:07.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:08.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:09.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:09.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:09.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:33:09.234 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:33:09.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:09.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:09.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:10.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:10.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:10.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:33:10.730 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:33:10.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:10.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:11.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:12.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:12.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:12.216 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:33:12.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:33:12.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:12.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:13.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:14.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:14.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:14.144 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:33:14.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:33:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:15.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:16.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:16.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:16.054 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:33:16.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:33:16.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:16.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:16.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:17.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:17.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:17.908 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:33:17.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:33:17.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:17.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:18.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:19.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:19.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:19.945 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:33:19.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:33:19.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:19.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:20.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:21.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:21.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:21.718 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:33:21.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:33:21.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:21.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:22.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:23.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:23.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:23.699 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:33:23.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:33:23.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:23.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:24.145 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4233214759,v1:127.0.0.1:6803/4233214759] [v2:127.0.0.1:6804/4233214759,v1:127.0.0.1:6805/4233214759] exists,up baf8edda-a00e-4bb1-be52-579a9eab7ed7 2024-04-25T08:33:24.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:33:24.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:33:24.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:33:24.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:33:24.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:33:24.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e 2024-04-25T08:33:24.152 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e 2024-04-25T08:33:24.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e' 2024-04-25T08:33:24.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:33:24.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUFSpm0DB4ChAAOKGEmi6LMUP06ZMbmsX06Q== 2024-04-25T08:33:24.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUFSpm0DB4ChAAOKGEmi6LMUP06ZMbmsX06Q=="}' 2024-04-25T08:33:24.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e -i td/osd-recovery-stats/1/new.json 2024-04-25T08:33:24.677 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:33:24.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:33:24.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUFSpm0DB4ChAAOKGEmi6LMUP06ZMbmsX06Q== --osd-uuid 3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e 2024-04-25T08:33:24.726 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.731+0000 7fd71c04a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:24.751 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.755+0000 7fd71c04a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:24.752 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.755+0000 7fd71c04a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:24.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.759+0000 7fd71c04a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:33:24.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.759+0000 7fd71c04a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:33:24.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.759+0000 7fd71c04a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:33:24.790 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:24.795+0000 7fd71c04a7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:33:31.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:33:31.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:33:31.827 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:33:31.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:33:31.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:33:32.400 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:33:32.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:33:32.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:33:32.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:33:32.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:33:32.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:33:32.433 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:32.439+0000 7f8aa23417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:32.435 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:32.439+0000 7f8aa23417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:32.436 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:32.439+0000 7f8aa23417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:32.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:33:32.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:33:32.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:33:32.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:33:32.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:33:32.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:32.809 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:33:32.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:33:32.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:32.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:33.011 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:33.015+0000 7f8aa23417c0 -1 Falling back to public interface 2024-04-25T08:33:33.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:34.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:34.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:34.242 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:33:34.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:33:34.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:34.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:34.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:35.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:35.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:35.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:33:35.665 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:33:35.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:35.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:35.816 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:33:35.819+0000 7f8aa23417c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:33:36.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:37.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:37.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:37.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:33:37.193 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:33:37.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:37.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:37.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:38.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:33:38.775 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:33:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:38.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:39.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:40.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:40.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:40.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:33:40.314 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:33:40.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:40.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:40.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:41.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:41.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:41.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:33:41.902 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:33:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:42.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:43.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:43.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:43.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:33:43.391 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:33:43.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:43.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:44.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:45.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:45.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:45.014 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:33:45.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:33:45.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:45.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:45.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:46.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:46.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:46.641 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:33:46.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:33:46.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:46.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:47.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:48.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:48.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:48.085 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:33:48.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:33:48.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:48.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:48.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:49.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:49.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:49.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:33:49.535 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:33:49.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:49.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:50.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:51.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:51.039 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:33:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:33:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:51.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:51.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:52.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:52.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:52.648 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:33:52.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:33:52.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:52.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:53.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:54.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:54.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:54.524 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:33:54.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:33:54.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:54.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:55.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:56.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:56.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:56.316 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:33:56.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:33:56.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:56.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:57.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:58.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:58.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:58.097 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:33:58.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:33:58.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:58.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:58.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:59.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:59.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:59.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:33:59.914 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:33:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:59.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:01.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:02.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:02.104 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:34:02.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:34:02.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:02.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:02.539 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/308951531,v1:127.0.0.1:6811/308951531] [v2:127.0.0.1:6812/308951531,v1:127.0.0.1:6813/308951531] exists,up 3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e 2024-04-25T08:34:02.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:34:02.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:34:02.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:34:02.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3adc6416-1f2e-437d-a509-b34f863c60d2 2024-04-25T08:34:02.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3adc6416-1f2e-437d-a509-b34f863c60d2' 2024-04-25T08:34:02.545 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 3adc6416-1f2e-437d-a509-b34f863c60d2 2024-04-25T08:34:02.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:34:02.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6FSpm/dwlIhAABCELFU3NhQaQgpHvtOlYGQ== 2024-04-25T08:34:02.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6FSpm/dwlIhAABCELFU3NhQaQgpHvtOlYGQ=="}' 2024-04-25T08:34:02.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3adc6416-1f2e-437d-a509-b34f863c60d2 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:34:03.250 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:34:03.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:34:03.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6FSpm/dwlIhAABCELFU3NhQaQgpHvtOlYGQ== --osd-uuid 3adc6416-1f2e-437d-a509-b34f863c60d2 2024-04-25T08:34:03.312 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.314+0000 7f50b25ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:03.315 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.318+0000 7f50b25ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:03.317 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.322+0000 7f50b25ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:03.317 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.322+0000 7f50b25ae7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:34:03.317 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.322+0000 7f50b25ae7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:34:03.317 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.322+0000 7f50b25ae7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:34:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:03.426+0000 7f50b25ae7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:34:10.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:34:10.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:34:10.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:34:10.202 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:34:10.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:34:10.785 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:34:10.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:34:10.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:34:10.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:34:10.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:34:10.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:34:10.819 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:10.822+0000 7fc70cc8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:10.831 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:10.834+0000 7fc70cc8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:10.832 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:10.834+0000 7fc70cc8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:11.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:34:11.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:34:11.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:34:11.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:34:11.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:34:11.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:11.232 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:34:11.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:34:11.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:11.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:11.386 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:11.390+0000 7fc70cc8c7c0 -1 Falling back to public interface 2024-04-25T08:34:11.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:12.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:12.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:12.674 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:34:12.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:34:12.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:12.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:13.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:14.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:14.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:14.101 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:34:14.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:34:14.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:14.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:14.138 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:14.142+0000 7fc70cc8c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:34:14.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:15.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:15.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:15.576 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:34:15.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:34:15.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:15.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:16.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:17.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:17.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:17.273 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:34:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:17.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:17.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:18.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:18.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:18.828 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:34:18.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:18.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:18.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:19.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:20.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:20.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:20.266 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:34:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:20.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:20.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:21.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:21.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:21.706 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:34:21.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:34:21.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:21.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:22.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:23.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:23.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:23.247 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:34:23.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:34:23.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:23.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:23.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:24.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:24.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:24.774 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:34:24.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:34:24.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:24.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:25.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:26.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:26.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:26.295 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:34:26.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:34:26.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:26.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:26.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:27.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:27.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:27.774 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:34:27.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:34:27.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:27.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:28.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:29.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:29.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:29.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:34:29.371 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:34:29.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:29.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:30.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:31.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:31.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:31.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:34:31.005 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:34:31.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:31.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:31.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:32.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:32.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:32.452 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:34:32.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:34:32.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:32.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:33.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:34.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:34.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:34.096 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:34:34.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:34:34.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:34.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:35.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:36.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:36.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:36.116 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:34:36.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:34:36.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:36.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:37.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:38.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:38.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:38.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:34:38.002 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:34:38.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:38.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:38.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:39.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:39.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:39.696 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:34:39.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:34:39.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:39.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:40.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:41.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:41.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:41.573 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T08:34:41.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:34:41.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:41.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:42.007 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2193662183,v1:127.0.0.1:6819/2193662183] [v2:127.0.0.1:6820/2193662183,v1:127.0.0.1:6821/2193662183] exists,up 3adc6416-1f2e-437d-a509-b34f863c60d2 2024-04-25T08:34:42.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:34:42.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:34:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:34:42.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:34:42.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cfe367eb-2425-408f-bf0e-bcc41db9be24 2024-04-25T08:34:42.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 cfe367eb-2425-408f-bf0e-bcc41db9be24' 2024-04-25T08:34:42.014 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 cfe367eb-2425-408f-bf0e-bcc41db9be24 2024-04-25T08:34:42.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:34:42.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiFSpmIIIeAhAAJVQx17QOKLFz8c869FZUhg== 2024-04-25T08:34:42.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiFSpmIIIeAhAAJVQx17QOKLFz8c869FZUhg=="}' 2024-04-25T08:34:42.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cfe367eb-2425-408f-bf0e-bcc41db9be24 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:34:42.555 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:34:42.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:34:42.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiFSpmIIIeAhAAJVQx17QOKLFz8c869FZUhg== --osd-uuid cfe367eb-2425-408f-bf0e-bcc41db9be24 2024-04-25T08:34:42.607 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.609+0000 7fb35522c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:42.636 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.637+0000 7fb35522c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:42.637 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.641+0000 7fb35522c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:42.638 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.641+0000 7fb35522c7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:34:42.638 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.641+0000 7fb35522c7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:34:42.638 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.641+0000 7fb35522c7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:34:42.675 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:42.677+0000 7fb35522c7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:34:49.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:34:49.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:34:49.703 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:34:49.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:34:49.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:34:50.286 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:34:50.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:34:50.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:34:50.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:34:50.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:34:50.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:34:50.320 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:50.321+0000 7f8e40cad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:50.340 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:50.341+0000 7f8e40cad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:50.341 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:50.345+0000 7f8e40cad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:50.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:34:50.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:34:50.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:34:50.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:34:50.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:34:50.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:50.735 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:34:50.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:34:50.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:50.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:50.923 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:50.925+0000 7f8e40cad7c0 -1 Falling back to public interface 2024-04-25T08:34:51.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:52.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:52.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:52.212 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:34:52.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:34:52.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:52.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:52.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:53.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:53.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:53.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:34:53.648 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:34:53.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:53.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:53.775 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:34:53.777+0000 7f8e40cad7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:34:54.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:55.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:55.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:55.162 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:34:55.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:34:55.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:55.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:55.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:56.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:56.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:56.788 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:34:56.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:56.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:56.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:57.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:58.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:58.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:58.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:58.346 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:34:58.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:58.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:58.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:59.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:59.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:59.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:59.868 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:34:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:59.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:00.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:01.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:01.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:01.304 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:35:01.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:35:01.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:01.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:01.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:02.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:02.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:02.798 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:35:02.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:35:02.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:02.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:03.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:04.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:04.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:04.338 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:35:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:35:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:04.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:04.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:05.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:05.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:05.848 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:35:05.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:35:05.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:05.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:06.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:07.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:07.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:07.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:35:07.399 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:35:07.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:07.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:08.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:09.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:09.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:09.079 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:35:09.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:35:09.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:09.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:09.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:10.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:10.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:10.634 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:35:10.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:35:10.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:10.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:11.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:12.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:12.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:12.690 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:35:12.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:35:12.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:12.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:13.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:14.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:14.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:14.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:35:14.831 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:35:14.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:14.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:15.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:16.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:16.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:16.683 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:35:16.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:35:16.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:16.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:17.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:18.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:18.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:18.857 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:35:18.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:35:18.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:18.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:19.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:20.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:20.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:20.937 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:35:20.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:35:20.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:20.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:21.410 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2748947977,v1:127.0.0.1:6827/2748947977] [v2:127.0.0.1:6828/2748947977,v1:127.0.0.1:6829/2748947977] exists,up cfe367eb-2425-408f-bf0e-bcc41db9be24 2024-04-25T08:35:21.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:35:21.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:35:21.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:35:21.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:35:21.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:35:21.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:35:21.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5bd002b-8a4d-4efe-8b4a-2d1b04af8207 2024-04-25T08:35:21.417 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 a5bd002b-8a4d-4efe-8b4a-2d1b04af8207 2024-04-25T08:35:21.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a5bd002b-8a4d-4efe-8b4a-2d1b04af8207' 2024-04-25T08:35:21.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:35:21.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJFSpmUh8tGhAAkTFXS8LBm3NiUl52h5jzZg== 2024-04-25T08:35:21.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJFSpmUh8tGhAAkTFXS8LBm3NiUl52h5jzZg=="}' 2024-04-25T08:35:21.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5bd002b-8a4d-4efe-8b4a-2d1b04af8207 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:35:21.930 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:35:21.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:35:21.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJFSpmUh8tGhAAkTFXS8LBm3NiUl52h5jzZg== --osd-uuid a5bd002b-8a4d-4efe-8b4a-2d1b04af8207 2024-04-25T08:35:21.987 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:21.989+0000 7f364307e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:21.989 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:21.993+0000 7f364307e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:21.991 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:21.993+0000 7f364307e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:21.991 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:21.993+0000 7f364307e7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:35:21.991 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:21.993+0000 7f364307e7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:35:21.991 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:21.993+0000 7f364307e7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:35:22.034 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:22.037+0000 7f364307e7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:35:29.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:35:29.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:35:29.031 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T08:35:29.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:35:29.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:35:29.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:35:29.605 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T08:35:29.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:35:29.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:35:29.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:35:29.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:35:29.638 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:29.641+0000 7fc8340297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:29.640 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:29.645+0000 7fc8340297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:29.641 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:29.645+0000 7fc8340297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:35:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:35:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:35:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:35:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:35:30.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:30.045 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:35:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:35:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:30.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:30.193 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:30.197+0000 7fc8340297c0 -1 Falling back to public interface 2024-04-25T08:35:30.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:31.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:31.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:31.488 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:35:31.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:35:31.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:31.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:31.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:32.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:35:32.829+0000 7fc8340297c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:35:32.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:35:32.914 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:35:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:33.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:34.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:34.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:34.645 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:35:34.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:35:34.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:34.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:35.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:36.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:36.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:36.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:35:36.153 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:35:36.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:36.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:36.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:37.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:37.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:37.698 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:35:37.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:35:37.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:37.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:38.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:39.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:39.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:39.208 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:35:39.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:35:39.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:39.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:39.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:40.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:40.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:40.813 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:35:40.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:35:40.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:40.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:41.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:42.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:42.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:42.327 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:35:42.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:35:42.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:42.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:42.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:43.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:43.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:43.866 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:35:43.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:35:43.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:43.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:44.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:45.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:45.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:45.307 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:35:45.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:35:45.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:45.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:45.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:46.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:46.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:46.783 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:35:46.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:35:46.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:46.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:47.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:48.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:48.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:48.324 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:35:48.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:35:48.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:48.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:49.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:50.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:50.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:50.191 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:35:50.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:35:50.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:50.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:51.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:52.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:52.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:52.100 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:35:52.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:35:52.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:52.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:53.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:54.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:54.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:54.112 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:35:54.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:35:54.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:54.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:54.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:55.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:55.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:55.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:35:55.773 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:35:55.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:55.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:56.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:57.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:57.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:57.448 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:35:57.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:35:57.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:57.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:58.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:59.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:59.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:59.435 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:35:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:35:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:59.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:59.921 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1565630161,v1:127.0.0.1:6835/1565630161] [v2:127.0.0.1:6836/1565630161,v1:127.0.0.1:6837/1565630161] exists,up a5bd002b-8a4d-4efe-8b4a-2d1b04af8207 2024-04-25T08:35:59.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:35:59.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:35:59.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:35:59.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:35:59.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:35:59.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd7da147-be75-4011-ba95-6a29839b3d63 2024-04-25T08:35:59.932 INFO:tasks.workunit.client.0.smithi031.stdout:add osd5 cd7da147-be75-4011-ba95-6a29839b3d63 2024-04-25T08:35:59.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 cd7da147-be75-4011-ba95-6a29839b3d63' 2024-04-25T08:35:59.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:35:59.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvFSpm0cAiORAAle+NA5g+GaQ6B8sGeyiWCg== 2024-04-25T08:35:59.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvFSpm0cAiORAAle+NA5g+GaQ6B8sGeyiWCg=="}' 2024-04-25T08:35:59.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd7da147-be75-4011-ba95-6a29839b3d63 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:36:00.539 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:36:00.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:36:00.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvFSpm0cAiORAAle+NA5g+GaQ6B8sGeyiWCg== --osd-uuid cd7da147-be75-4011-ba95-6a29839b3d63 2024-04-25T08:36:00.584 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.584+0000 7f37b73fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:00.616 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.620+0000 7f37b73fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:00.618 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.620+0000 7f37b73fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:00.618 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.620+0000 7f37b73fb7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:36:00.618 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.620+0000 7f37b73fb7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:36:00.619 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.620+0000 7f37b73fb7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:36:00.663 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:00.664+0000 7f37b73fb7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:36:07.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:36:07.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:36:07.599 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd5 key to auth repository 2024-04-25T08:36:07.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:36:07.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:36:08.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:36:08.169 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.5 2024-04-25T08:36:08.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:36:08.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:36:08.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:36:08.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:36:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:08.204+0000 7f448b9537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:08.231 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:08.232+0000 7f448b9537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:08.233 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:08.236+0000 7f448b9537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:08.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:36:08.627 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:36:08.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:08.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:08.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:08.812+0000 7f448b9537c0 -1 Falling back to public interface 2024-04-25T08:36:09.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:10.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:10.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:10.114 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:36:10.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:36:10.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:10.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:10.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:11.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:11.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:11.543 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:36:11.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:36:11.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:11.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:11.649 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:36:11.652+0000 7f448b9537c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:36:12.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:13.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:13.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:13.090 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:36:13.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:36:13.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:13.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:13.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:14.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:14.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:14.702 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:36:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:36:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:15.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:16.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:16.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:16.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:36:16.181 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:36:16.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:16.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:16.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:17.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:17.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:17.675 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:36:17.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:36:17.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:17.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:18.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:19.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:19.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:19.151 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:36:19.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:36:19.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:19.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:19.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:20.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:20.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:20.712 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:36:20.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:36:20.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:20.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:21.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:22.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:22.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:22.215 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:36:22.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:36:22.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:22.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:22.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:23.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:23.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:23.775 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:36:23.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:36:23.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:23.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:24.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:25.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:25.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:25.264 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:36:25.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:36:25.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:25.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:25.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:26.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:26.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:26.798 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:36:26.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:36:26.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:26.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:27.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:28.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:28.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:28.543 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:36:28.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:36:28.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:28.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:29.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:30.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:30.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:30.355 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:36:30.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:36:30.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:30.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:31.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:32.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:32.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:32.219 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:36:32.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:36:32.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:32.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:33.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:34.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:34.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:34.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:36:34.117 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:36:34.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:34.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:34.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:35.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:35.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:35.798 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:36:35.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:36:35.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:35.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:36.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:37.854 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:36:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:36:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:38.370 INFO:tasks.workunit.client.0.smithi031.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2105586852,v1:127.0.0.1:6843/2105586852] [v2:127.0.0.1:6844/2105586852,v1:127.0.0.1:6845/2105586852] exists,up cd7da147-be75-4011-ba95-6a29839b3d63 2024-04-25T08:36:38.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:36:38.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:36:38.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:36:38.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-04-25T08:36:38.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:36:39.151 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:36:39.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:36:40.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-04-25T08:36:41.500 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 3 2024-04-25T08:36:41.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-04-25T08:36:42.522 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 1 2024-04-25T08:36:42.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-04-25T08:36:42.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:36:42.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:36:42.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:36:42.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:36:42.547 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:36:42.547 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:36:42.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:36:42.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:36:42.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:36:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:36:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:36:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:36:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:36:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:36:42.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:36:43.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:43.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:36:43.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T08:36:43.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T08:36:43.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T08:36:43.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:43.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:36:43.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673001 2024-04-25T08:36:43.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673001 2024-04-25T08:36:43.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001' 2024-04-25T08:36:43.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:43.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:36:43.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509469 2024-04-25T08:36:43.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509469 2024-04-25T08:36:43.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509469' 2024-04-25T08:36:43.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:43.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:36:43.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T08:36:43.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T08:36:43.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509469 3-85899345942' 2024-04-25T08:36:43.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:43.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:36:44.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182416 2024-04-25T08:36:44.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182416 2024-04-25T08:36:44.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509469 3-85899345942 4-107374182416' 2024-04-25T08:36:44.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:44.018 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:36:44.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018895 2024-04-25T08:36:44.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018895 2024-04-25T08:36:44.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509469 3-85899345942 4-107374182416 5-128849018895' 2024-04-25T08:36:44.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:44.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T08:36:44.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:44.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:36:44.172 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T08:36:44.172 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:44.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T08:36:44.173 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836527 2024-04-25T08:36:44.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T08:36:44.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:36:44.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836527 2024-04-25T08:36:44.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:36:45.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:36:45.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:36:46.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836528 -lt 21474836527 2024-04-25T08:36:46.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:46.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673001 2024-04-25T08:36:46.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:46.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:36:46.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673001 2024-04-25T08:36:46.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:46.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673001 2024-04-25T08:36:46.022 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673001 2024-04-25T08:36:46.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-04-25T08:36:46.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:36:46.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673001 -lt 42949673001 2024-04-25T08:36:46.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:46.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509469 2024-04-25T08:36:46.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:46.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:36:46.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509469 2024-04-25T08:36:46.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509469 2024-04-25T08:36:46.448 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509469 2024-04-25T08:36:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-04-25T08:36:46.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:36:46.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509469 2024-04-25T08:36:46.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:46.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345942 2024-04-25T08:36:46.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:46.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:36:46.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345942 2024-04-25T08:36:46.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:46.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T08:36:46.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-04-25T08:36:46.891 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345942 2024-04-25T08:36:46.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:36:47.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345943 -lt 85899345942 2024-04-25T08:36:47.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:47.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182416 2024-04-25T08:36:47.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:47.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:36:47.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182416 2024-04-25T08:36:47.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:47.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182416 2024-04-25T08:36:47.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2024-04-25T08:36:47.321 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182416 2024-04-25T08:36:47.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:36:47.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182417 -lt 107374182416 2024-04-25T08:36:47.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:47.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018895 2024-04-25T08:36:47.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:47.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:36:47.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018895 2024-04-25T08:36:47.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:47.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018895 2024-04-25T08:36:47.769 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018895 2024-04-25T08:36:47.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018895' 2024-04-25T08:36:47.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:36:48.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018895 -lt 128849018895 2024-04-25T08:36:48.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:36:48.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:48.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:48.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:36:48.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:36:48.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:36:48.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:36:48.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:36:48.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:36:48.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:36:48.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:36:49.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:36:49.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:36:49.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:49.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:36:49.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:36:49.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:36:49.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-04-25T08:36:50.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:36:50.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:36:50.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:36:50.037 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:36:50.037 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:36:50.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-04-25T08:36:50.476 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-04-25T08:36:50.476 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:36:50.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:36:50.477 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:36:50.477 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:36:50.477 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:36:50.477 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:36:50.478 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:36:50.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:36:50.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:36:50.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:36:51.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-04-25T08:36:51.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-04-25T08:36:52.055 INFO:tasks.workunit.client.0.smithi031.stderr:noout is set 2024-04-25T08:36:52.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-04-25T08:36:52.934 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:36:52.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:36:52.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 305327 2024-04-25T08:36:52.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-04-25T08:36:53.842 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T08:36:53.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-04-25T08:36:53.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-04-25T08:36:53.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-04-25T08:36:53.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:53.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-04-25T08:36:54.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:54.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-04-25T08:36:54.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:54.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-04-25T08:36:54.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:54.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-04-25T08:36:54.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:54.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-04-25T08:36:54.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:54.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-04-25T08:36:54.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:54.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-04-25T08:36:55.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-04-25T08:36:55.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-04-25T08:36:55.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-04-25T08:36:55.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-04-25T08:36:55.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-04-25T08:36:55.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-04-25T08:36:55.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:55.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-04-25T08:36:56.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-04-25T08:36:56.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-04-25T08:36:56.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-04-25T08:36:56.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-04-25T08:36:56.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-04-25T08:36:56.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-04-25T08:36:56.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:56.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-04-25T08:36:57.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-04-25T08:36:57.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-04-25T08:36:57.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-04-25T08:36:57.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-04-25T08:36:57.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-04-25T08:36:57.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-04-25T08:36:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-04-25T08:36:58.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-04-25T08:36:58.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-04-25T08:36:58.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-04-25T08:36:58.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-04-25T08:36:58.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-04-25T08:36:58.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-04-25T08:36:58.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-04-25T08:36:58.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:58.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-04-25T08:36:59.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:59.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-04-25T08:36:59.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:59.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-04-25T08:36:59.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:59.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-04-25T08:36:59.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:59.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-04-25T08:36:59.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:59.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-04-25T08:36:59.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:36:59.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-04-25T08:37:00.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-04-25T08:37:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-04-25T08:37:00.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-04-25T08:37:00.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-04-25T08:37:00.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-04-25T08:37:00.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-04-25T08:37:00.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-04-25T08:37:00.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:00.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-04-25T08:37:01.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-04-25T08:37:01.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-04-25T08:37:01.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-04-25T08:37:01.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-04-25T08:37:01.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-04-25T08:37:01.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-04-25T08:37:01.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:01.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-04-25T08:37:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-04-25T08:37:02.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-04-25T08:37:02.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-04-25T08:37:02.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-04-25T08:37:02.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-04-25T08:37:02.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-04-25T08:37:02.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-04-25T08:37:02.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:02.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-04-25T08:37:03.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:03.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-04-25T08:37:03.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:03.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-04-25T08:37:03.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:03.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-04-25T08:37:03.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:03.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-04-25T08:37:03.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:03.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-04-25T08:37:03.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:03.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-04-25T08:37:04.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-04-25T08:37:04.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-04-25T08:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-04-25T08:37:04.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-04-25T08:37:04.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-04-25T08:37:04.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-04-25T08:37:04.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-04-25T08:37:04.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:04.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-04-25T08:37:05.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-04-25T08:37:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-04-25T08:37:05.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-04-25T08:37:05.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-04-25T08:37:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-04-25T08:37:05.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-04-25T08:37:05.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:05.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-04-25T08:37:06.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-04-25T08:37:06.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-04-25T08:37:06.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-04-25T08:37:06.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-04-25T08:37:06.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-04-25T08:37:06.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-04-25T08:37:06.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:06.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-04-25T08:37:07.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:07.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-04-25T08:37:07.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:07.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-04-25T08:37:07.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:07.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-04-25T08:37:07.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:07.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-04-25T08:37:07.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:07.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-04-25T08:37:07.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:37:07.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-04-25T08:37:07.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-04-25T08:37:07.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 307090 2024-04-25T08:37:07.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-04-25T08:37:08.665 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T08:37:08.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-04-25T08:37:08.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:37:08.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:37:08.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:08.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:08.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:08.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:08.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:37:08.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:37:08.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:37:08.693 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:37:08.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:37:08.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:37:08.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-04-25T08:37:08.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:37:08.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:37:08.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:37:08.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:37:08.720 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:08.727+0000 7f93501767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:08.729 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:08.735+0000 7f93501767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:08.731 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:08.735+0000 7f93501767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:09.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:37:09.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:37:09.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:37:09.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:37:09.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:37:09.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:09.219 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:37:09.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:37:09.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:09.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:09.281 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:09.287+0000 7f93501767c0 -1 Falling back to public interface 2024-04-25T08:37:09.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:10.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:10.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:10.720 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:37:10.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:37:10.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:10.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:11.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:12.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:12.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:12.161 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:37:12.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:37:12.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:12.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:12.442 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:12.447+0000 7f93501767c0 -1 osd.0 38 log_to_monitors true 2024-04-25T08:37:12.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:13.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:13.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:13.586 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:37:13.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:37:13.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:13.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:14.032 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 43 last_clean_interval [5,38) [v2:127.0.0.1:6802/208148591,v1:127.0.0.1:6803/208148591] [v2:127.0.0.1:6804/208148591,v1:127.0.0.1:6805/208148591] exists,up baf8edda-a00e-4bb1-be52-579a9eab7ed7 2024-04-25T08:37:14.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:37:14.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:37:14.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:37:14.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-04-25T08:37:17.036 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-04-25T08:37:17.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-04-25T08:37:17.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-04-25T08:37:17.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-04-25T08:37:17.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-04-25T08:37:17.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-04-25T08:37:17.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-04-25T08:37:17.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-04-25T08:37:17.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-04-25T08:37:17.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:17.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-04-25T08:37:18.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-04-25T08:37:18.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-04-25T08:37:18.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-04-25T08:37:18.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-04-25T08:37:18.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-04-25T08:37:18.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-04-25T08:37:18.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-04-25T08:37:18.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:18.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-04-25T08:37:19.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-04-25T08:37:19.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-04-25T08:37:19.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-04-25T08:37:19.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-04-25T08:37:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-04-25T08:37:19.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-04-25T08:37:19.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:19.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-04-25T08:37:20.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-04-25T08:37:20.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-04-25T08:37:20.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-04-25T08:37:20.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-04-25T08:37:20.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-04-25T08:37:20.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-04-25T08:37:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-04-25T08:37:20.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:20.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-04-25T08:37:21.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-04-25T08:37:21.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-04-25T08:37:21.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-04-25T08:37:21.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-04-25T08:37:21.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-04-25T08:37:21.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-04-25T08:37:21.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:21.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-04-25T08:37:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-04-25T08:37:22.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-04-25T08:37:22.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-04-25T08:37:22.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-04-25T08:37:22.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-04-25T08:37:22.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-04-25T08:37:22.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-04-25T08:37:22.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:22.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-04-25T08:37:23.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-04-25T08:37:23.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-04-25T08:37:23.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-04-25T08:37:23.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-04-25T08:37:23.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-04-25T08:37:23.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-04-25T08:37:23.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:23.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-04-25T08:37:24.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-04-25T08:37:24.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-04-25T08:37:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-04-25T08:37:24.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-04-25T08:37:24.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-04-25T08:37:24.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-04-25T08:37:24.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-04-25T08:37:24.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:24.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-04-25T08:37:25.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-04-25T08:37:25.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-04-25T08:37:25.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-04-25T08:37:25.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-04-25T08:37:25.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-04-25T08:37:25.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-04-25T08:37:25.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-04-25T08:37:25.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:25.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-04-25T08:37:26.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-04-25T08:37:26.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-04-25T08:37:26.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-04-25T08:37:26.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-04-25T08:37:26.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-04-25T08:37:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-04-25T08:37:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:26.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-04-25T08:37:27.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-04-25T08:37:27.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-04-25T08:37:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-04-25T08:37:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-04-25T08:37:27.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-04-25T08:37:27.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-04-25T08:37:27.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:27.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-04-25T08:37:28.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-04-25T08:37:28.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-04-25T08:37:28.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-04-25T08:37:28.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-04-25T08:37:28.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-04-25T08:37:28.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-04-25T08:37:28.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-04-25T08:37:28.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:28.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-04-25T08:37:29.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-04-25T08:37:29.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-04-25T08:37:29.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-04-25T08:37:29.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-04-25T08:37:29.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-04-25T08:37:29.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-04-25T08:37:29.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:29.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-04-25T08:37:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:30.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-04-25T08:37:30.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:37:30.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-04-25T08:37:30.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-04-25T08:37:30.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:37:30.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:37:30.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:37:30.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:37:30.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-04-25T08:37:30.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-04-25T08:37:30.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:37:30.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-04-25T08:37:30.790 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-04-25T08:37:30.790 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:37:30.790 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-04-25T08:37:30.791 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-04-25T08:37:30.791 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:37:31.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-04-25T08:37:31.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-04-25T08:37:31.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-04-25T08:37:31.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-04-25T08:37:31.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-04-25T08:37:32.452 INFO:tasks.workunit.client.0.smithi031.stderr:noout is unset 2024-04-25T08:37:32.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-04-25T08:37:33.183 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already out. osd.2 is already out. 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:37:33.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:37:33.203 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:37:33.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T08:37:33.206 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:37:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:37:33.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-04-25T08:37:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T08:37:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:37:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:37:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:37:33.230 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:33.235+0000 7f13781647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:33.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:33.239+0000 7f13781647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:33.233 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:33.239+0000 7f13781647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:33.770 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:33.774+0000 7f13781647c0 -1 Falling back to public interface 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:33.921 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:37:33.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:37:33.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:33.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:34.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:35.651 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:37:35.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:37:35.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:35.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:36.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:37.051 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:37:37.054+0000 7f13781647c0 -1 osd.1 44 log_to_monitors true 2024-04-25T08:37:37.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:37.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:37.065 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:37:37.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:37:37.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:37.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:37.659 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up out weight 0 up_from 57 up_thru 43 down_at 45 last_clean_interval [10,44) [v2:127.0.0.1:6810/72357267,v1:127.0.0.1:6811/72357267] [v2:127.0.0.1:6812/72357267,v1:127.0.0.1:6813/72357267] exists,up 3d74114e-fbcb-4fc5-88b0-5a22b3ec7c2e 2024-04-25T08:37:37.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:37:37.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:37:37.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:37:37.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-04-25T08:37:40.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-04-25T08:37:41.866 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 4 2024-04-25T08:37:41.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-04-25T08:37:43.065 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:37:43.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:37:43.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:37:43.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:37:43.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:37:43.088 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:37:43.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T08:37:43.855 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:37:43.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-04-25T08:37:45.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-04-25T08:37:45.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:37:45.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:37:45.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:37:45.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:37:45.874 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:37:45.874 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:37:45.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:37:45.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:37:45.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:37:46.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:37:46.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:37:46.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:37:46.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:37:46.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:37:46.004 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:37:46.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:37:46.449 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:37:46.449 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:37:46.449 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:37:46.449 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:37:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:37:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:37:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:46.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:37:46.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=206158430217 2024-04-25T08:37:46.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 206158430217 2024-04-25T08:37:46.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430217' 2024-04-25T08:37:46.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:46.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:37:46.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=244813135876 2024-04-25T08:37:46.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 244813135876 2024-04-25T08:37:46.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430217 1-244813135876' 2024-04-25T08:37:46.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:46.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:37:46.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509484 2024-04-25T08:37:46.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509484 2024-04-25T08:37:46.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430217 1-244813135876 2-64424509484' 2024-04-25T08:37:46.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:46.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:37:47.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345957 2024-04-25T08:37:47.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345957 2024-04-25T08:37:47.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430217 1-244813135876 2-64424509484 3-85899345957' 2024-04-25T08:37:47.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:47.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:37:47.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182431 2024-04-25T08:37:47.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182431 2024-04-25T08:37:47.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430217 1-244813135876 2-64424509484 3-85899345957 4-107374182431' 2024-04-25T08:37:47.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:47.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:37:47.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018909 2024-04-25T08:37:47.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018909 2024-04-25T08:37:47.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430217 1-244813135876 2-64424509484 3-85899345957 4-107374182431 5-128849018909' 2024-04-25T08:37:47.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:47.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-206158430217 2024-04-25T08:37:47.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:47.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:37:47.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-206158430217 2024-04-25T08:37:47.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:47.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=206158430217 2024-04-25T08:37:47.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 206158430217' 2024-04-25T08:37:47.529 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 206158430217 2024-04-25T08:37:47.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:37:47.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 206158430217 -lt 206158430217 2024-04-25T08:37:47.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:47.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-244813135876 2024-04-25T08:37:47.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:47.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:37:47.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-244813135876 2024-04-25T08:37:47.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:47.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=244813135876 2024-04-25T08:37:47.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 244813135876' 2024-04-25T08:37:47.991 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 244813135876 2024-04-25T08:37:47.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:37:48.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 244813135876 -lt 244813135876 2024-04-25T08:37:48.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:48.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509484 2024-04-25T08:37:48.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:48.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:37:48.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509484 2024-04-25T08:37:48.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:48.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509484 2024-04-25T08:37:48.504 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509484 2024-04-25T08:37:48.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-04-25T08:37:48.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:37:49.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509484 -lt 64424509484 2024-04-25T08:37:49.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:49.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345957 2024-04-25T08:37:49.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:49.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:37:49.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345957 2024-04-25T08:37:49.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:49.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345957 2024-04-25T08:37:49.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345957' 2024-04-25T08:37:49.043 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345957 2024-04-25T08:37:49.043 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:37:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345957 -lt 85899345957 2024-04-25T08:37:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:49.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182431 2024-04-25T08:37:49.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:49.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:37:49.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:49.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182431 2024-04-25T08:37:49.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182431 2024-04-25T08:37:49.649 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182431 2024-04-25T08:37:49.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2024-04-25T08:37:49.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:37:50.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182431 -lt 107374182431 2024-04-25T08:37:50.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:50.239 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018909 2024-04-25T08:37:50.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:50.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:37:50.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018909 2024-04-25T08:37:50.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:50.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018909 2024-04-25T08:37:50.244 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018909 2024-04-25T08:37:50.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018909' 2024-04-25T08:37:50.244 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:37:50.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018910 -lt 128849018909 2024-04-25T08:37:50.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:37:50.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:50.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:51.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:37:51.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:51.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:51.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:51.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:51.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:51.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:51.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:51.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:51.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:51.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:51.977 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:52.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:52.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:37:52.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:37:52.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:37:52.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:52.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:52.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:52.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:52.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:52.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:52.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:52.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:52.633 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:52.634 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:53.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:53.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:53.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:53.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:53.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:53.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:53.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:54.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:37:54.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:37:54.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:54.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:54.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:54.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:54.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:54.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:54.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:54.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:54.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:54.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:54.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:55.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:55.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:55.017 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:55.017 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:55.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:55.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:55.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:55.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:55.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:37:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:37:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:56.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:56.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:56.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:56.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:56.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:56.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:56.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:56.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:56.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:56.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:56.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:56.820 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:57.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:57.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:57.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:58.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:37:58.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:37:58.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:58.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:58.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:58.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:58.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:58.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:58.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:58.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:58.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:58.142 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:58.142 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:58.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:58.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:58.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:58.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:59.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:59.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:37:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:37:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:59.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:59.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:59.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:59.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:59.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:59.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:59.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:59.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:00.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:00.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:00.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:00.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:01.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:01.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:01.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:01.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:01.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:01.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:01.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:01.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:01.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:01.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:38:01.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:38:01.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:01.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:01.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:01.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:01.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:01.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:01.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:01.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:01.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:01.786 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:02.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:02.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:02.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:03.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:38:03.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:38:03.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:03.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:03.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:03.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:03.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:03.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:03.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:03.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:03.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:03.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:03.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:04.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:04.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:04.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:04.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:04.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:04.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:04.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:05.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:38:05.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:38:05.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:05.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:05.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:05.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:05.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:05.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:05.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:05.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:05.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:05.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:05.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:05.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:05.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:05.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:06.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:06.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:06.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:07.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:38:07.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:38:07.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:07.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:07.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:07.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:07.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:07.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:07.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:07.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:07.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:07.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:07.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:07.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:07.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:07.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:07.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:08.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:08.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:38:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:38:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:09.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:09.208 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:09.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:09.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:09.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:09.732 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:10.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:10.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:11.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:38:11.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:38:11.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:11.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:11.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:11.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:11.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:11.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:11.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:11.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:11.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:11.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:11.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:11.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:11.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:11.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:11.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:12.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:12.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:12.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:12.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:12.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:12.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:12.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:12.510 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:12.510 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:13.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:38:13.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:38:13.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:13.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:13.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:13.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:13.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:13.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:13.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:13.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:13.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:13.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:13.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:13.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:13.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:13.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:13.701 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:14.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:14.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:15.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:38:15.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:38:15.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:15.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:15.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:15.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:15.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:15.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:15.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:15.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:15.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:15.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:15.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:15.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:15.698 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:15.698 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:15.699 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:16.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:16.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:16.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:16.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:16.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:16.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:16.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:16.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:16.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:17.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:38:17.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:38:17.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:17.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:17.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:17.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:17.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:17.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:17.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:17.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:17.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:17.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:17.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:17.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:17.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:17.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:17.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:18.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:18.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:18.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:18.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:18.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:18.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:18.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:18.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:18.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:18.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:38:18.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:38:18.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:18.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:18.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:19.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:19.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:19.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:19.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:19.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:19.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:19.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:19.079 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:19.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:38:19.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:19.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:19.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:20.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:38:20.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:38:20.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:38:20.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:38:20.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:38:20.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:38:20.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:38:20.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:38:20.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:38:20.529 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:38:20.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-04-25T08:38:20.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:38:20.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:20.564 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.564 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:38:20.564 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:38:20.564 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:38:20.564 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:38:20.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:20.570 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.570 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:38:20.570 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:20.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:38:20.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:38:20.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:38:20.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:38:20.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:38:20.582 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:38:20.587 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:20.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:38:20.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:38:20.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:38:20.599 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:38:20.601 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:20.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-04-25T08:38:20.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-04-25T08:38:20.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-04-25T08:38:20.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:38:20.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-04-25T08:38:20.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:38:20.604 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:20.604 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:38:20.604 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:38:20.604 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:38:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:38:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:38:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:38:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:38:20.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:38:20.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-04-25T08:38:20.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:38:20.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:38:21.134 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:38:21.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-04-25T08:38:21.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:38:21.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:38:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:38:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:38:21.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:38:22.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:38:22.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:38:22.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:38:22.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:38:22.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:38:22.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:38:22.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:38:22.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:38:22.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:38:22.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:38:22.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:38:22.476 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:38:22.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:38:22.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:38:22.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:38:22.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:38:22.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:38:22.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:22.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:38:22.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:38:22.480 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:22.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:38:22.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:38:22.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:38:22.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:38:22.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:22.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:22.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:38:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:38:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:38:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:38:22.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:38:22.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:38:22.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:38:22.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:38:22.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:38:22.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:38:22.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:38:22.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:38:22.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:38:22.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:22.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:38:22.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:38:22.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:22.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:38:22.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:38:22.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:38:22.538 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:38:22.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:22.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:22.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:38:22.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:38:22.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:38:22.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:38:22.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:38:22.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:22.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:22.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:38:22.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:38:22.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:38:22.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:38:22.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:38:23.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:38:23.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:23.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:23.012 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:23.012 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:23.012 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:23.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:38:23.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:38:23.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:38:23.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:38:23.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:38:23.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:38:23.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:38:23.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:38:23.242 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:38:23.242 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:38:23.242 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:38:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:38:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:38:23.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:38:23.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:38:23.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:38:23.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:38:23.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:38:23.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:38:23.365 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:38:23.366 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:38:23.366 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:38:23.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:38:23.366 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:38:23.366 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:23.367 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:23.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:38:23.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:38:23.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:38:23.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:38:23.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:38:23.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:38:23.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:38:23.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:38:23.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:38:23.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:38:24.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:38:24.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:24.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:24.414 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:24.414 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:24.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:24.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:38:24.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:38:24.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:38:24.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:38:24.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:38:24.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:38:24.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:38:24.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:24.455 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:38:24.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:38:24.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:38:24.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:38:24.457 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:38:24.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb35c137-52bb-4555-aeb9-09c0cd7ec5d3 2024-04-25T08:38:24.457 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 eb35c137-52bb-4555-aeb9-09c0cd7ec5d3 2024-04-25T08:38:24.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eb35c137-52bb-4555-aeb9-09c0cd7ec5d3' 2024-04-25T08:38:24.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:38:24.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAFipm07zPHBAAYDUUUIY6FEt+2zMHhkpSFQ== 2024-04-25T08:38:24.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAFipm07zPHBAAYDUUUIY6FEt+2zMHhkpSFQ=="}' 2024-04-25T08:38:24.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb35c137-52bb-4555-aeb9-09c0cd7ec5d3 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:38:24.800 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:38:24.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:38:24.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAFipm07zPHBAAYDUUUIY6FEt+2zMHhkpSFQ== --osd-uuid eb35c137-52bb-4555-aeb9-09c0cd7ec5d3 2024-04-25T08:38:24.859 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.862+0000 7fa9133c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:24.877 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.882+0000 7fa9133c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:24.878 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.882+0000 7fa9133c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:24.879 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.882+0000 7fa9133c27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:38:24.879 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.882+0000 7fa9133c27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:38:24.879 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.882+0000 7fa9133c27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:38:24.917 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:24.922+0000 7fa9133c27c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:38:31.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:38:31.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:38:31.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:38:31.540 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:38:31.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:38:32.105 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:38:32.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:38:32.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:38:32.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:38:32.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:38:32.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:38:32.139 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:32.141+0000 7fab8ad367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:32.165 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:32.169+0000 7fab8ad367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:32.167 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:32.169+0000 7fab8ad367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:32.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:38:32.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:38:32.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:38:32.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:38:32.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:38:32.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:32.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:38:32.581 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:38:32.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:32.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:32.721 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:32.725+0000 7fab8ad367c0 -1 Falling back to public interface 2024-04-25T08:38:33.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:34.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:34.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:34.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:34.060 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:38:34.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:34.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:34.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:35.346 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:38:35.349+0000 7fab8ad367c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:38:35.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:35.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:35.507 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:38:35.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:35.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:35.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:36.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:37.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:37.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:37.116 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:38:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:37.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:38.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:38.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:38.589 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:38:38.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:38:38.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:38.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:39.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:40.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:40.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:40.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:38:40.024 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:38:40.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:40.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:40.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:41.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:41.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:41.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:38:41.633 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:38:41.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:41.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:42.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:43.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:43.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:43.076 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:38:43.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:38:43.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:43.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:43.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:44.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:44.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:44.535 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:38:44.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:38:44.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:44.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:45.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:46.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:46.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:46.022 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:38:46.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:38:46.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:46.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:46.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:47.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:47.555 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:38:47.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:47.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:38:47.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:47.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:48.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:49.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:49.007 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:38:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:38:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:49.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:50.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:50.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:50.986 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:38:50.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:38:50.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:50.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:51.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:52.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:52.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:52.484 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:38:52.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:38:52.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:52.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:53.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:54.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:54.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:54.313 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:38:54.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:38:54.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:54.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:55.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:56.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:56.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:56.065 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:38:56.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:38:56.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:56.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:56.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:57.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:57.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:57.764 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:38:57.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:38:57.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:57.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:58.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:59.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:59.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:59.949 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:38:59.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:38:59.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:59.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:39:00.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:01.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:01.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:01.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:01.441 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:39:01.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:01.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:39:01.961 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2692742345,v1:127.0.0.1:6803/2692742345] [v2:127.0.0.1:6804/2692742345,v1:127.0.0.1:6805/2692742345] exists,up eb35c137-52bb-4555-aeb9-09c0cd7ec5d3 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:01.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:39:01.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:01.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:39:01.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:01.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71f9fd5c-0220-4714-8eb7-bae9cfbd30e2 2024-04-25T08:39:01.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 71f9fd5c-0220-4714-8eb7-bae9cfbd30e2' 2024-04-25T08:39:01.968 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 71f9fd5c-0220-4714-8eb7-bae9cfbd30e2 2024-04-25T08:39:01.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:01.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClFipmapuBOxAA718/Mvq95B5Lg2e9/FWPuQ== 2024-04-25T08:39:01.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClFipmapuBOxAA718/Mvq95B5Lg2e9/FWPuQ=="}' 2024-04-25T08:39:01.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71f9fd5c-0220-4714-8eb7-bae9cfbd30e2 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:39:02.493 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:39:02.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:39:02.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClFipmapuBOxAA718/Mvq95B5Lg2e9/FWPuQ== --osd-uuid 71f9fd5c-0220-4714-8eb7-bae9cfbd30e2 2024-04-25T08:39:02.549 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.553+0000 7f641745b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:02.592 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.597+0000 7f641745b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:02.593 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.597+0000 7f641745b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:02.594 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.597+0000 7f641745b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:39:02.594 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.597+0000 7f641745b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:39:02.594 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.597+0000 7f641745b7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:39:02.638 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:02.641+0000 7f641745b7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:39:09.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:39:09.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:09.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:39:09.309 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:39:09.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:09.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:39:09.890 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:39:09.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:09.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:09.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:09.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:09.923 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:09.925+0000 7f1158a437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:09.934 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:09.937+0000 7f1158a437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:09.936 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:09.941+0000 7f1158a437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:10.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:39:10.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:10.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:39:10.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:10.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:10.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:10.358 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:39:10.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:10.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:10.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:10.509 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:10.513+0000 7f1158a437c0 -1 Falling back to public interface 2024-04-25T08:39:10.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:11.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:11.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:11.831 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:39:11.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:11.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:11.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:12.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:13.092 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:13.097+0000 7f1158a437c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:39:13.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:13.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:13.258 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:39:13.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:39:13.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:13.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:13.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:14.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:14.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:14.851 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:39:14.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:39:14.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:14.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:15.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:16.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:16.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:16.289 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:39:16.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:39:16.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:16.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:16.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:17.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:17.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:17.737 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:39:17.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:39:17.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:17.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:18.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:19.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:19.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:19.318 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:39:19.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:39:19.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:19.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:20.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:20.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:20.958 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:39:20.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:39:20.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:20.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:21.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:22.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:22.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:22.399 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:39:22.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:39:22.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:22.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:23.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:24.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:24.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:24.012 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:39:24.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:39:24.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:24.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:24.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:25.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:25.552 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:39:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:39:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:25.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:25.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:26.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:26.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:26.987 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:39:26.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:39:26.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:26.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:27.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:28.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:28.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:28.723 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:39:28.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:39:28.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:28.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:29.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:30.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:30.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:30.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:39:30.559 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:39:30.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:30.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:31.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:32.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:32.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:32.290 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:39:32.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:39:32.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:32.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:32.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:33.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:33.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:33.863 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:39:33.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:33.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:33.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:34.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:35.565 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:39:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:36.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:37.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:37.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:37.120 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:39:37.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:39:37.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:37.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:37.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:38.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:38.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:38.872 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:39:38.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:38.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:38.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:39.426 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3062455076,v1:127.0.0.1:6811/3062455076] [v2:127.0.0.1:6812/3062455076,v1:127.0.0.1:6813/3062455076] exists,up 71f9fd5c-0220-4714-8eb7-bae9cfbd30e2 2024-04-25T08:39:39.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:39.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:39:39.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:39:39.430 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:39:39.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:39:39.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:39.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:39:39.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:39.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f50a9424-2494-4bf0-b6c4-36b57e54c638 2024-04-25T08:39:39.433 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 f50a9424-2494-4bf0-b6c4-36b57e54c638 2024-04-25T08:39:39.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f50a9424-2494-4bf0-b6c4-36b57e54c638' 2024-04-25T08:39:39.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLFipmQoltGxAAq90BUoONjamhkPisdErxug== 2024-04-25T08:39:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLFipmQoltGxAAq90BUoONjamhkPisdErxug=="}' 2024-04-25T08:39:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f50a9424-2494-4bf0-b6c4-36b57e54c638 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:39:40.010 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:39:40.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:39:40.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLFipmQoltGxAAq90BUoONjamhkPisdErxug== --osd-uuid f50a9424-2494-4bf0-b6c4-36b57e54c638 2024-04-25T08:39:40.065 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.068+0000 7fbcb06777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:40.096 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.100+0000 7fbcb06777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:40.098 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.100+0000 7fbcb06777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:40.098 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.100+0000 7fbcb06777c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:39:40.098 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.100+0000 7fbcb06777c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:39:40.099 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.100+0000 7fbcb06777c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:39:40.143 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:40.148+0000 7fbcb06777c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:39:47.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:39:47.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:47.082 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:39:47.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:39:47.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:47.652 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:39:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:39:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:47.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:47.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:47.688+0000 7f583f7a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:47.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:47.720+0000 7f583f7a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:47.719 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:47.720+0000 7f583f7a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:39:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:39:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:48.115 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:39:48.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:48.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:48.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:48.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:48.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:48.286 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:48.288+0000 7f583f7a47c0 -1 Falling back to public interface 2024-04-25T08:39:48.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:49.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:49.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:49.575 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:39:49.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:49.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:49.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:50.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:50.988 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:39:50.992+0000 7f583f7a47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:39:51.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:51.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:51.051 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:39:51.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:39:51.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:51.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:51.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:52.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:52.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:52.672 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:39:52.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:39:52.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:52.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:53.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:54.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:54.137 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:39:54.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:39:54.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:54.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:54.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:55.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:55.656 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:39:55.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:55.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:39:55.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:55.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:56.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:57.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:57.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:57.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:39:57.251 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:39:57.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:57.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:57.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:58.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:58.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:58.896 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:39:58.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:39:58.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:58.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:59.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:00.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:00.337 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:40:00.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:40:00.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:00.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:00.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:01.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:01.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:01.778 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:40:01.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:40:01.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:01.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:02.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:03.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:03.231 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:40:03.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:03.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:40:03.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:03.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:03.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:04.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:04.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:04.828 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:40:04.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:40:04.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:04.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:05.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:06.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:06.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:06.340 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:40:06.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:40:06.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:06.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:07.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:08.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:08.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:08.368 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:40:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:40:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:09.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:10.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:10.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:10.200 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:40:10.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:40:10.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:10.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:11.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:12.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:12.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:12.132 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:40:12.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:40:12.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:12.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:13.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:14.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:14.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:14.234 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:40:14.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:40:14.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:14.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:14.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:15.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:15.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:15.959 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:40:15.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:40:15.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:15.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:16.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:17.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:17.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:17.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:40:17.858 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:40:17.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:17.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:18.295 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3615387683,v1:127.0.0.1:6819/3615387683] [v2:127.0.0.1:6820/3615387683,v1:127.0.0.1:6821/3615387683] exists,up f50a9424-2494-4bf0-b6c4-36b57e54c638 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:40:18.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:40:18.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:40:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:40:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:40:18.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:40:18.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a906987f-ba99-4b12-9d1a-3a44f7ddd8ec 2024-04-25T08:40:18.302 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 a906987f-ba99-4b12-9d1a-3a44f7ddd8ec 2024-04-25T08:40:18.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a906987f-ba99-4b12-9d1a-3a44f7ddd8ec' 2024-04-25T08:40:18.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:40:18.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyFipmkNDDExAAQ/j6QJr5vmKIaxK9Z53AwQ== 2024-04-25T08:40:18.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyFipmkNDDExAAQ/j6QJr5vmKIaxK9Z53AwQ=="}' 2024-04-25T08:40:18.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a906987f-ba99-4b12-9d1a-3a44f7ddd8ec -i td/osd-recovery-stats/3/new.json 2024-04-25T08:40:18.995 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:40:19.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:40:19.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyFipmkNDDExAAQ/j6QJr5vmKIaxK9Z53AwQ== --osd-uuid a906987f-ba99-4b12-9d1a-3a44f7ddd8ec 2024-04-25T08:40:19.050 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.055+0000 7f2ed4d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:19.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.059+0000 7f2ed4d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:19.055 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.063+0000 7f2ed4d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:19.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.063+0000 7f2ed4d417c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:40:19.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.063+0000 7f2ed4d417c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:40:19.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.063+0000 7f2ed4d417c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:40:19.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:19.107+0000 7f2ed4d417c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:40:26.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:40:26.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:40:26.171 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:40:26.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:40:26.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:40:26.746 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:40:26.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:40:26.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:40:26.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:40:26.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:40:26.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:40:26.771 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:26.779+0000 7fda5f40c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:26.774 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:26.779+0000 7fda5f40c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:26.775 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:26.783+0000 7fda5f40c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:27.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:40:27.206 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:40:27.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:27.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:27.314 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:27.319+0000 7fda5f40c7c0 -1 Falling back to public interface 2024-04-25T08:40:27.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:28.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:28.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:28.657 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:40:28.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:40:28.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:28.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:29.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:30.073 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:30.079+0000 7fda5f40c7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:40:30.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:30.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:30.095 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:40:30.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:40:30.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:30.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:30.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:31.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:31.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:31.812 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:40:31.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:40:31.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:31.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:32.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:33.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:33.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:33.285 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:40:33.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:40:33.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:33.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:33.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:34.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:34.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:34.819 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:40:34.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:40:34.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:34.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:35.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:36.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:36.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:36.312 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:40:36.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:40:36.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:36.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:36.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:37.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:37.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:37.746 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:40:37.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:40:37.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:37.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:38.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:39.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:39.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:39.341 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:40:39.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:40:39.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:39.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:39.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:40.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:40.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:40.827 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:40:40.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:40:40.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:40.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:41.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:42.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:42.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:42.311 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:40:42.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:40:42.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:42.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:42.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:43.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:43.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:43.781 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:40:43.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:40:43.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:43.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:45.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:45.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:45.310 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:40:45.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:40:45.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:45.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:46.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:47.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:47.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:47.012 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:40:47.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:40:47.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:47.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:47.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:48.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:48.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:48.852 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:40:48.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:40:48.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:48.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:50.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:51.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:51.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:51.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:40:51.094 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:40:51.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:51.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:52.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:53.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:53.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:53.125 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:40:53.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:40:53.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:53.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:53.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:54.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:54.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:54.939 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:40:54.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:40:54.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:54.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:55.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:56.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:56.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:56.662 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:40:56.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:40:56.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:56.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:57.106 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1761598444,v1:127.0.0.1:6827/1761598444] [v2:127.0.0.1:6828/1761598444,v1:127.0.0.1:6829/1761598444] exists,up a906987f-ba99-4b12-9d1a-3a44f7ddd8ec 2024-04-25T08:40:57.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:40:57.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:40:57.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:40:57.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:40:57.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:40:57.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:40:57.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:40:57.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:40:57.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1a13f74-0b60-4407-ba90-49d143d8a879 2024-04-25T08:40:57.113 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 d1a13f74-0b60-4407-ba90-49d143d8a879 2024-04-25T08:40:57.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d1a13f74-0b60-4407-ba90-49d143d8a879' 2024-04-25T08:40:57.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:40:57.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZFypmKXR6CBAAPSoVSwvR0svA8i+3zlYMFQ== 2024-04-25T08:40:57.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZFypmKXR6CBAAPSoVSwvR0svA8i+3zlYMFQ=="}' 2024-04-25T08:40:57.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1a13f74-0b60-4407-ba90-49d143d8a879 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:40:57.842 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:40:57.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:40:57.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZFypmKXR6CBAAPSoVSwvR0svA8i+3zlYMFQ== --osd-uuid d1a13f74-0b60-4407-ba90-49d143d8a879 2024-04-25T08:40:57.898 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:57.903+0000 7f683a73b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:57.900 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:57.907+0000 7f683a73b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:57.901 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:57.907+0000 7f683a73b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:57.902 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:57.907+0000 7f683a73b7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:40:57.902 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:57.907+0000 7f683a73b7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:40:57.902 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:57.907+0000 7f683a73b7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:40:58.000 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:40:58.007+0000 7f683a73b7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:41:05.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:41:05.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:41:05.033 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T08:41:05.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:41:05.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:41:05.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:41:05.600 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T08:41:05.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:41:05.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:41:05.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:41:05.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:41:05.633 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:05.639+0000 7f0ded7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:05.661 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:05.667+0000 7f0ded7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:05.663 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:05.671+0000 7f0ded7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:06.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:41:06.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:41:06.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:06.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:06.237 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:06.243+0000 7f0ded7dc7c0 -1 Falling back to public interface 2024-04-25T08:41:06.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:07.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:07.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:07.499 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:41:07.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:41:07.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:07.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:07.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:08.913 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:08.919+0000 7f0ded7dc7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:41:08.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:08.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:08.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:41:08.952 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:41:08.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:08.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:09.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:10.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:10.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:10.752 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:41:10.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:41:10.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:10.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:11.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:12.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:12.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:12.197 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:41:12.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:41:12.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:12.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:12.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:13.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:13.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:13.644 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:41:13.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:41:13.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:13.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:14.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:15.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:15.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:15.270 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:41:15.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:41:15.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:15.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:15.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:16.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:16.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:16.897 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:41:16.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:41:16.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:16.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:17.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:18.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:18.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:18.338 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:41:18.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:41:18.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:18.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:18.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:19.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:19.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:19.857 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:41:19.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:41:19.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:19.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:20.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:21.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:21.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:21.309 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:41:21.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:41:21.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:21.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:22.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:23.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:23.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:23.018 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:41:23.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:41:23.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:23.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:23.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:24.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:24.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:24.742 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:41:24.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:41:24.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:24.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:25.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:26.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:26.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:26.617 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:41:26.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:41:26.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:26.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:27.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:28.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:28.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:28.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:41:28.374 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:41:28.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:28.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:29.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:30.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:30.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:30.160 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:41:30.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:41:30.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:30.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:30.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:31.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:31.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:31.709 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:41:31.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:41:31.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:31.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:32.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:33.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:33.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:33.654 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:41:33.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:41:33.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:33.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:35.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:35.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:35.529 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:41:35.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:41:35.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:35.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:35.976 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2157949256,v1:127.0.0.1:6835/2157949256] [v2:127.0.0.1:6836/2157949256,v1:127.0.0.1:6837/2157949256] exists,up d1a13f74-0b60-4407-ba90-49d143d8a879 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:41:35.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:41:35.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:41:35.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:41:35.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:41:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9c954b8-0dbb-4dec-b137-8622832778c9 2024-04-25T08:41:35.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 e9c954b8-0dbb-4dec-b137-8622832778c9' 2024-04-25T08:41:35.983 INFO:tasks.workunit.client.0.smithi031.stdout:add osd5 e9c954b8-0dbb-4dec-b137-8622832778c9 2024-04-25T08:41:35.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:41:36.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAFypmOkzEABAABrVUQ40KVRItLs10FEf++g== 2024-04-25T08:41:36.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAFypmOkzEABAABrVUQ40KVRItLs10FEf++g=="}' 2024-04-25T08:41:36.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9c954b8-0dbb-4dec-b137-8622832778c9 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:41:36.668 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:41:36.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:41:36.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAFypmOkzEABAABrVUQ40KVRItLs10FEf++g== --osd-uuid e9c954b8-0dbb-4dec-b137-8622832778c9 2024-04-25T08:41:36.724 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.730+0000 7f3c4a0877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:36.727 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.734+0000 7f3c4a0877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:36.728 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.734+0000 7f3c4a0877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:36.729 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.734+0000 7f3c4a0877c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:41:36.729 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.734+0000 7f3c4a0877c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:41:36.729 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.734+0000 7f3c4a0877c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:41:36.772 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:36.778+0000 7f3c4a0877c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:41:43.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:41:43.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:41:43.713 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd5 key to auth repository 2024-04-25T08:41:43.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:41:43.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:41:44.291 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.5 2024-04-25T08:41:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:41:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:41:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:41:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:41:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:41:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:44.330+0000 7f3aab83f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:44.357 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:44.362+0000 7f3aab83f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:44.359 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:44.366+0000 7f3aab83f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:44.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:41:44.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:41:44.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:41:44.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:41:44.743 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:41:44.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:41:44.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:44.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:41:44.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:44.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:44.924 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:44.930+0000 7f3aab83f7c0 -1 Falling back to public interface 2024-04-25T08:41:45.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:46.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:46.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:46.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:41:46.177 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:41:46.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:46.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:46.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:47.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:47.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:47.599 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:41:47.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:41:47.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:47.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:47.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:41:47.742+0000 7f3aab83f7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:41:48.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:49.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:49.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:49.122 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:41:49.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:41:49.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:49.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:49.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:50.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:50.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:50.576 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:41:50.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:41:50.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:50.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:51.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:52.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:52.175 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:41:52.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:41:52.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:52.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:52.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:53.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:53.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:53.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:41:53.767 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:41:53.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:53.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:54.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:55.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:55.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:55.256 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:41:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:41:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:55.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:56.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:56.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:56.867 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:41:56.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:41:56.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:56.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:57.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:58.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:58.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:58.314 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:41:58.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:41:58.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:58.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:58.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:59.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:59.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:59.826 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:41:59.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:41:59.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:59.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:00.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:01.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:01.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:01.306 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:42:01.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:42:01.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:01.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:01.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:02.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:02.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:02.830 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:42:02.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:42:02.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:02.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:03.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:04.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:04.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:04.475 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:42:04.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:42:04.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:04.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:05.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:06.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:06.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:06.822 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:42:06.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:42:06.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:06.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:07.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:08.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:08.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:08.751 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:42:08.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:42:08.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:08.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:09.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:10.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:10.534 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:42:10.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:42:10.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:10.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:11.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:12.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:12.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:12.464 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:42:12.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:42:12.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:12.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:13.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:14.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:14.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:14.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:42:14.276 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:42:14.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:14.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:14.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:15.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:15.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:15.785 INFO:tasks.workunit.client.0.smithi031.stdout:19 2024-04-25T08:42:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:42:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:15.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:16.300 INFO:tasks.workunit.client.0.smithi031.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2358576283,v1:127.0.0.1:6843/2358576283] [v2:127.0.0.1:6844/2358576283,v1:127.0.0.1:6845/2358576283] exists,up e9c954b8-0dbb-4dec-b137-8622832778c9 2024-04-25T08:42:16.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:42:16.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:42:16.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:42:16.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-04-25T08:42:16.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-04-25T08:42:16.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-04-25T08:42:16.979 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:42:17.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:42:18.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:42:18.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:42:18.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:42:18.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:42:18.004 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:42:18.004 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:42:18.005 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:42:18.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:42:18.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:42:18.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:42:18.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:42:18.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:42:18.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:42:18.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:42:18.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:42:18.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:42:18.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:42:18.572 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:42:18.572 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:42:18.572 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:42:18.572 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:42:18.573 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:42:18.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:42:18.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:18.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:42:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-04-25T08:42:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-04-25T08:42:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-04-25T08:42:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:18.749 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:42:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673000 2024-04-25T08:42:18.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673000 2024-04-25T08:42:18.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000' 2024-04-25T08:42:18.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:18.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:42:19.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509472 2024-04-25T08:42:19.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509472 2024-04-25T08:42:19.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509472' 2024-04-25T08:42:19.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:19.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:42:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T08:42:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T08:42:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509472 3-85899345941' 2024-04-25T08:42:19.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:19.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:42:19.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182415 2024-04-25T08:42:19.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182415 2024-04-25T08:42:19.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509472 3-85899345941 4-107374182415' 2024-04-25T08:42:19.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:19.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:42:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018886 2024-04-25T08:42:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018886 2024-04-25T08:42:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509472 3-85899345941 4-107374182415 5-128849018886' 2024-04-25T08:42:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:19.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-04-25T08:42:19.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:42:19.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:19.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-04-25T08:42:19.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-04-25T08:42:19.688 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836526 2024-04-25T08:42:19.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-04-25T08:42:19.688 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:42:20.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836526 -lt 21474836526 2024-04-25T08:42:20.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:20.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673000 2024-04-25T08:42:20.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:20.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:42:20.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673000 2024-04-25T08:42:20.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:20.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673000 2024-04-25T08:42:20.138 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673000 2024-04-25T08:42:20.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-04-25T08:42:20.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:42:20.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673000 -lt 42949673000 2024-04-25T08:42:20.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:20.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509472 2024-04-25T08:42:20.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:20.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:42:20.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509472 2024-04-25T08:42:20.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:20.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509472 2024-04-25T08:42:20.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-04-25T08:42:20.577 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509472 2024-04-25T08:42:20.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:42:21.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509472 -lt 64424509472 2024-04-25T08:42:21.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:21.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345941 2024-04-25T08:42:21.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:21.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:42:21.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-04-25T08:42:21.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:21.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T08:42:21.012 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345941 2024-04-25T08:42:21.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-04-25T08:42:21.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:42:21.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345941 2024-04-25T08:42:21.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:21.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182415 2024-04-25T08:42:21.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:21.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:42:21.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182415 2024-04-25T08:42:21.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:21.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182415 2024-04-25T08:42:21.467 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182415 2024-04-25T08:42:21.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-04-25T08:42:21.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:42:21.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182416 -lt 107374182415 2024-04-25T08:42:21.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:21.897 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018886 2024-04-25T08:42:21.897 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:21.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:42:21.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018886 2024-04-25T08:42:21.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:21.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018886 2024-04-25T08:42:21.901 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018886 2024-04-25T08:42:21.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-04-25T08:42:21.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:42:22.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018887 -lt 128849018886 2024-04-25T08:42:22.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:42:22.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:22.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:22.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:22.904 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:23.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:42:23.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:23.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:23.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:23.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:42:23.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:42:23.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:42:23.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:42:23.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:23.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:42:24.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:24.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:42:24.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:24.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:42:24.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:24.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:42:24.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:24.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:42:24.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:24.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:42:24.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:24.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:42:25.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:42:25.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:42:25.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:42:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:42:25.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:42:25.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:42:25.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:25.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:42:26.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:42:26.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:42:26.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:42:26.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:42:26.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:42:26.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:42:26.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:26.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:42:27.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:27.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:42:27.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:27.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:42:27.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:27.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:42:27.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:27.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:42:27.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:27.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:42:27.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:27.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:42:28.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:42:28.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:42:28.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:42:28.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:42:28.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:42:28.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:42:28.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:28.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:42:29.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:29.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:42:29.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:29.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:42:29.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:29.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:42:29.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:29.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:42:29.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:29.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:42:29.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:29.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:42:30.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:42:30.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:42:30.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:42:30.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:42:30.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:42:30.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:42:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:42:31.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:31.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:42:31.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:31.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:42:31.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:31.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:42:31.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:31.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:42:31.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:31.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:42:31.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:31.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:42:32.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:42:32.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:42:32.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:42:32.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:42:32.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:42:32.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:42:32.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:32.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:42:33.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:33.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:42:33.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:33.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:42:33.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:33.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:42:33.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:33.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:42:33.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:33.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:42:33.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:33.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:42:34.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:42:34.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:42:34.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:42:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:42:34.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:42:34.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:42:34.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:34.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:42:35.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:35.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:42:35.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:35.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:42:35.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:35.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:42:35.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:35.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:42:35.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:35.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:42:35.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:35.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:42:36.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:36.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:42:36.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:36.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:42:36.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:36.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:42:36.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:36.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:42:36.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:36.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:42:36.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:36.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:42:37.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:37.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:42:37.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:37.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:42:37.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:37.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:42:37.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:37.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:42:37.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:37.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:42:37.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:37.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:42:38.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:42:38.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:42:38.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:42:38.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:42:38.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:42:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:42:38.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:38.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:42:39.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:42:39.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:42:39.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:42:39.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:42:39.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:42:39.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:42:39.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:39.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:42:40.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:40.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:42:40.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:40.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:42:40.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:40.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:42:40.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:40.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:42:40.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:40.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:42:40.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:40.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:42:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:41.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:42:41.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:41.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:42:41.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:41.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:42:41.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:41.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:42:41.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:41.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:42:41.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:41.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:42:42.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:42:42.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:42:42.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:42:42.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:42:42.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:42:42.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:42:42.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:42.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:42:43.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:43.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:42:43.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:43.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:42:43.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:43.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:42:43.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:43.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:42:43.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:43.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:42:43.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:43.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:42:44.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:44.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:42:44.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:44.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:42:44.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:44.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:42:44.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:44.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:42:44.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:44.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:42:44.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:44.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:42:45.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:45.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:42:45.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:45.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:42:45.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:45.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:42:45.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:45.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:42:45.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:45.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:42:45.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:45.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:42:46.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:42:46.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:42:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:42:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:42:46.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:42:46.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:42:46.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:46.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:42:47.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:47.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:42:47.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:47.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:42:47.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:47.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:42:47.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:47.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:42:47.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:47.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:42:47.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:47.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:42:48.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:42:48.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:42:48.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:42:48.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:42:48.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:42:48.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:42:48.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:48.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:42:49.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:49.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:42:49.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:49.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:42:49.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:49.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:42:49.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:49.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:42:49.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:49.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:42:49.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:49.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:42:50.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:42:50.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:42:50.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:42:50.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:42:50.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:42:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:42:50.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:50.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:42:51.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:51.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:42:51.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:51.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:42:51.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:51.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:42:51.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:51.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:42:51.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:51.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:42:51.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:51.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:42:52.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:42:52.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:42:52.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:42:52.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:42:52.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:42:52.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:42:52.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:52.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:42:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:42:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:53.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:42:53.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:53.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:42:53.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:53.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:42:53.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:53.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:42:54.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:54.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:42:54.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:54.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:42:54.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:54.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:42:54.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:54.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:42:54.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:54.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:42:55.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:55.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:42:55.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:42:55.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:42:55.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:42:55.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:55.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:55.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:55.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:55.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:42:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:42:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:42:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:42:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:42:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:42:56.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:42:56.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:42:56.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:42:56.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:42:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:42:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:56.279 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:56.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:42:56.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:42:56.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:42:57.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:42:57.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:42:57.968 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:42:57.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:42:57.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 323994 2024-04-25T08:42:57.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:42:58.487 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T08:42:58.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:42:59.530 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T08:42:59.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:43:00.439 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:43:00.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:43:00.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:43:00.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:43:00.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:43:00.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:43:01.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:43:01.216 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:43:01.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:43:03.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:43:03.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:43:03.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:43:03.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:43:03.234 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:43:03.235 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:43:03.235 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:43:03.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:43:03.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:43:03.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:43:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:43:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:43:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:43:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:43:03.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:43:03.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:43:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:43:03.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:43:04.008 INFO:tasks.workunit.client.0.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:43:04.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:43:04.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:43:04.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:43:04.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:43:04.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:43:04.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673011 2024-04-25T08:43:04.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673011 2024-04-25T08:43:04.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011' 2024-04-25T08:43:04.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:43:04.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:43:04.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509483 2024-04-25T08:43:04.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509483 2024-04-25T08:43:04.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509483' 2024-04-25T08:43:04.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:43:04.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:43:04.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345952 2024-04-25T08:43:04.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345952 2024-04-25T08:43:04.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509483 3-85899345952' 2024-04-25T08:43:04.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:43:04.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:43:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182426 2024-04-25T08:43:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182426 2024-04-25T08:43:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509483 3-85899345952 4-107374182426' 2024-04-25T08:43:04.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:43:04.612 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:43:04.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018897 2024-04-25T08:43:04.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018897 2024-04-25T08:43:04.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509483 3-85899345952 4-107374182426 5-128849018897' 2024-04-25T08:43:04.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:43:04.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673011 2024-04-25T08:43:04.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:43:04.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:43:04.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673011 2024-04-25T08:43:04.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:43:04.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673011 2024-04-25T08:43:04.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-04-25T08:43:04.764 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673011 2024-04-25T08:43:04.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:43:05.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673009 -lt 42949673011 2024-04-25T08:43:05.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:43:06.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:43:06.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:43:06.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673012 -lt 42949673011 2024-04-25T08:43:06.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:43:06.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509483 2024-04-25T08:43:06.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:43:06.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:43:06.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509483 2024-04-25T08:43:06.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:43:06.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509483 2024-04-25T08:43:06.618 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509483 2024-04-25T08:43:06.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2024-04-25T08:43:06.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:43:07.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509483 -lt 64424509483 2024-04-25T08:43:07.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:43:07.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345952 2024-04-25T08:43:07.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:43:07.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:43:07.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345952 2024-04-25T08:43:07.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:43:07.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345952 2024-04-25T08:43:07.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2024-04-25T08:43:07.136 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345952 2024-04-25T08:43:07.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:43:07.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345953 -lt 85899345952 2024-04-25T08:43:07.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:43:07.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182426 2024-04-25T08:43:07.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:43:07.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:43:07.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182426 2024-04-25T08:43:07.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:43:07.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182426 2024-04-25T08:43:07.685 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182426 2024-04-25T08:43:07.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2024-04-25T08:43:07.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:43:08.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182427 -lt 107374182426 2024-04-25T08:43:08.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:43:08.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018897 2024-04-25T08:43:08.154 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:43:08.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:43:08.156 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018897 2024-04-25T08:43:08.157 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:43:08.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018897 2024-04-25T08:43:08.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2024-04-25T08:43:08.158 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018897 2024-04-25T08:43:08.158 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:43:08.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018898 -lt 128849018897 2024-04-25T08:43:08.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:43:08.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:08.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:09.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:43:09.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:09.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:09.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:09.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:09.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:09.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:09.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:09.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:09.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:09.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:09.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:43:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:43:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:43:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:10.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:10.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:10.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:10.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:10.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:10.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:10.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:10.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:10.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:11.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:11.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:11.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:11.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:11.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:11.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:11.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:12.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:43:12.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:43:12.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:12.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:12.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:12.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:12.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:12.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:12.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:12.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:12.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:12.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:12.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:12.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:12.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:12.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:12.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:13.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:14.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:43:14.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:43:14.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:14.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:14.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:14.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:14.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:14.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:14.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:14.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:14.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:14.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:14.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:14.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:14.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:14.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:14.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:15.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:15.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:15.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:15.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:15.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:16.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-04-25T08:43:16.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-04-25T08:43:16.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:16.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:16.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:16.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:16.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:16.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:16.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:16.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:16.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:16.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:16.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:16.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:16.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:16.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:16.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:17.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:17.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:18.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17 2024-04-25T08:43:18.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17 '!=' null 2024-04-25T08:43:18.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:18.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:18.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:18.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:18.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:18.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:18.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:18.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:18.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:18.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:18.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:18.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:18.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:18.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:18.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:19.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:19.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:19.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:19.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:19.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:19.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:19.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:19.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:19.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:43:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:43:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:20.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:20.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:20.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:20.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:20.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:20.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:20.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:20.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:20.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:20.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:20.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:20.975 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:20.975 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:21.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:43:21.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:43:21.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:21.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:21.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:21.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:21.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:21.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:21.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:21.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:21.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:21.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:21.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:22.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:43:22.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:22.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:22.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:22.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:43:22.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:43:22.650 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:43:22.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-04-25T08:43:22.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:43:22.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:22.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:43:22.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:43:22.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:43:22.667 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:43:22.667 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:22.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:43:22.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:43:22.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:43:22.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:43:22.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:43:22.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:43:22.675 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:22.675 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:43:22.675 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:43:22.675 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:43:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:43:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:43:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:43:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:43:22.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:43:22.684 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:22.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:22.690 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:43:22.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:43:22.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:22.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:43:22.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:43:22.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:43:22.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:43:22.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:43:22.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:43:22.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:43:22.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:43:22.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:43:22.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:43:23.233 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:43:23.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:43:23.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:43:23.252 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:43:23.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:43:23.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:43:23.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:43:28.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:43:28.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:43:28.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:43:28.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:43:28.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:43:28.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:43:28.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:43:28.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:43:28.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:43:28.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:43:28.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:43:28.584 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:43:28.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:43:28.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:43:28.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:43:28.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:43:28.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:43:28.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:28.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:43:28.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:43:28.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:28.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:43:28.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:43:28.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:43:28.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:43:28.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:28.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:28.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:43:28.634 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:43:28.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:43:28.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:43:28.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:43:28.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:43:28.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:43:28.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:43:28.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:43:28.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:43:28.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:43:28.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:43:28.638 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:43:28.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:28.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:43:28.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:43:28.640 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:28.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:43:28.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:43:28.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:43:28.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:43:28.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:28.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:28.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:43:28.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:43:28.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:43:28.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:43:28.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:43:28.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:28.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:28.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-04-25T08:43:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:43:28.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:43:28.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:43:28.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:43:28.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:43:28.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:43:29.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:43:29.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:29.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:29.107 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:29.107 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:29.107 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:29.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:43:29.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:43:29.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:43:29.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:43:29.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:43:29.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:43:29.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:43:29.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:43:29.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:43:29.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:43:29.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:43:29.304 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:43:29.304 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:29.304 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:29.304 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:43:29.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:43:29.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:43:29.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:43:29.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:43:29.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:43:29.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:43:29.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:43:29.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:43:29.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-04-25T08:43:29.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:43:29.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:43:29.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:43:29.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:43:29.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:43:29.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:43:30.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:43:30.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:30.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:30.451 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:30.451 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:30.451 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:30.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:43:30.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:43:30.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:43:30.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:30.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:43:30.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:43:30.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:43:30.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:43:30.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ca3b8d0-b99d-4225-b29d-cc6372e53f85 2024-04-25T08:43:30.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1ca3b8d0-b99d-4225-b29d-cc6372e53f85' 2024-04-25T08:43:30.492 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 1ca3b8d0-b99d-4225-b29d-cc6372e53f85 2024-04-25T08:43:30.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:43:30.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyFypmgGXLHhAAZxa79hiz/dZvyF0bCqvOhA== 2024-04-25T08:43:30.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyFypmgGXLHhAAZxa79hiz/dZvyF0bCqvOhA=="}' 2024-04-25T08:43:30.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ca3b8d0-b99d-4225-b29d-cc6372e53f85 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:43:30.820 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:43:30.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:43:30.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyFypmgGXLHhAAZxa79hiz/dZvyF0bCqvOhA== --osd-uuid 1ca3b8d0-b99d-4225-b29d-cc6372e53f85 2024-04-25T08:43:30.874 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.876+0000 7ff8ea4a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:30.880 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.884+0000 7ff8ea4a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:30.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.884+0000 7ff8ea4a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:30.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.884+0000 7ff8ea4a27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:30.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.884+0000 7ff8ea4a27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:30.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.884+0000 7ff8ea4a27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:30.920 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:30.924+0000 7ff8ea4a27c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:43:37.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:43:37.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:43:37.041 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:43:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:43:37.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:43:37.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:43:37.619 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:43:37.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:43:37.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:43:37.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:43:37.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:43:37.644 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:37.648+0000 7f4c33cab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:37.654 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:37.656+0000 7f4c33cab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:37.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:37.660+0000 7f4c33cab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:38.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:43:38.057 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:43:38.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:38.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:38.191 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:38.196+0000 7f4c33cab7c0 -1 Falling back to public interface 2024-04-25T08:43:38.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:39.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:39.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:39.497 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:43:39.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:43:39.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:39.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:39.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:40.669 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:43:40.672+0000 7f4c33cab7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:43:40.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:40.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:40.926 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:43:40.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:43:40.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:40.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:41.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:42.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:42.397 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:43:42.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:42.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:43:42.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:42.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:42.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:43.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:43.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:43.894 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:43:43.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:43:43.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:43.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:44.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:45.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:45.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:45.379 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:43:45.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:43:45.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:45.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:45.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:46.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:46.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:46.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:43:46.824 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:43:46.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:46.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:47.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:48.432 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:43:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:43:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:48.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:48.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:49.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:49.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:49.906 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:43:49.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:43:49.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:49.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:50.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:51.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:51.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:51.370 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:43:51.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:43:51.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:51.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:52.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:53.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:53.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:53.138 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:43:53.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:43:53.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:53.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:53.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:54.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:54.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:54.733 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:43:54.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:43:54.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:54.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:55.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:56.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:56.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:56.366 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:43:56.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:43:56.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:56.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:58.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:58.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:58.247 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:43:58.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:43:58.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:58.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:58.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:59.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:59.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:59.883 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:43:59.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:43:59.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:59.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:01.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:01.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:01.575 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:44:01.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:01.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:01.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:02.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:03.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:03.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:03.046 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:44:03.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:44:03.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:03.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:03.505 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1994716139,v1:127.0.0.1:6803/1994716139] [v2:127.0.0.1:6804/1994716139,v1:127.0.0.1:6805/1994716139] exists,up 1ca3b8d0-b99d-4225-b29d-cc6372e53f85 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:44:03.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:44:03.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:44:03.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:44:03.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:44:03.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:44:03.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:44:03.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:44:03.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:44:03.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:44:03.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:44:03.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:44:03.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:44:03.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:44:03.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:44:03.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=325e9427-9df9-4850-a28b-678ad192f090 2024-04-25T08:44:03.514 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 325e9427-9df9-4850-a28b-678ad192f090 2024-04-25T08:44:03.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 325e9427-9df9-4850-a28b-678ad192f090' 2024-04-25T08:44:03.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:44:03.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTFypmANnsHxAAlOTX2aK62+CYYGiA//L3TA== 2024-04-25T08:44:03.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTFypmANnsHxAAlOTX2aK62+CYYGiA//L3TA=="}' 2024-04-25T08:44:03.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 325e9427-9df9-4850-a28b-678ad192f090 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:44:04.073 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:44:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:44:04.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTFypmANnsHxAAlOTX2aK62+CYYGiA//L3TA== --osd-uuid 325e9427-9df9-4850-a28b-678ad192f090 2024-04-25T08:44:04.131 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.140+0000 7f6b2afe87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:04.146 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.152+0000 7f6b2afe87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:04.147 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.156+0000 7f6b2afe87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.156+0000 7f6b2afe87c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:44:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.156+0000 7f6b2afe87c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:44:04.148 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.156+0000 7f6b2afe87c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:44:04.238 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:04.244+0000 7f6b2afe87c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:44:10.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:44:10.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:44:10.077 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:44:10.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:44:10.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:44:10.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:44:10.647 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:44:10.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:44:10.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:44:10.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:44:10.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:44:10.673 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:10.679+0000 7fa6977ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:10.692 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:10.699+0000 7fa6977ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:10.693 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:10.699+0000 7fa6977ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:11.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:44:11.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:44:11.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:44:11.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:44:11.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:44:11.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:11.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:44:11.099 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:44:11.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:11.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:11.264 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:11.271+0000 7fa6977ad7c0 -1 Falling back to public interface 2024-04-25T08:44:11.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:12.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:12.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:12.538 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:44:12.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:44:12.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:12.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:13.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:13.917 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:13.923+0000 7fa6977ad7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:44:14.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:14.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:14.028 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:44:14.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:44:14.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:14.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:14.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:15.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:15.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:15.587 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:44:15.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:44:15.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:15.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:16.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:17.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:17.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:17.150 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:44:17.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:44:17.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:17.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:17.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:18.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:18.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:18.607 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:44:18.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:44:18.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:18.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:19.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:20.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:20.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:20.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:44:20.026 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:44:20.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:20.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:20.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:21.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:21.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:21.505 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:44:21.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:44:21.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:21.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:22.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:23.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:23.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:23.095 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:44:23.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:44:23.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:23.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:23.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:24.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:24.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:24.548 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:44:24.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:24.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:24.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:24.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:25.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:25.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:25.989 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:44:25.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:44:25.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:25.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:26.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:27.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:27.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:27.726 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:44:27.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:44:27.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:27.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:28.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:29.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:29.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:29.708 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:44:29.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:44:29.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:29.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:30.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:31.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:31.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:31.413 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:44:31.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:44:31.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:31.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:32.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:33.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:33.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:33.190 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:44:33.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:44:33.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:33.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:34.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:35.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:35.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:35.631 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:44:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:36.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:37.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:37.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:37.081 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:44:37.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:44:37.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:37.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:37.561 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1435273400,v1:127.0.0.1:6811/1435273400] [v2:127.0.0.1:6812/1435273400,v1:127.0.0.1:6813/1435273400] exists,up 325e9427-9df9-4850-a28b-678ad192f090 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:44:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:44:37.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:44:37.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:44:37.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:44:37.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:44:37.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:44:37.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:44:37.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c66115a-0ff3-4f0b-bf30-b602e450a8a8 2024-04-25T08:44:37.567 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 5c66115a-0ff3-4f0b-bf30-b602e450a8a8 2024-04-25T08:44:37.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5c66115a-0ff3-4f0b-bf30-b602e450a8a8' 2024-04-25T08:44:37.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:44:37.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1Fypm8PFWIxAAIxbQ3oKUsROGUKueQmtIJA== 2024-04-25T08:44:37.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1Fypm8PFWIxAAIxbQ3oKUsROGUKueQmtIJA=="}' 2024-04-25T08:44:37.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c66115a-0ff3-4f0b-bf30-b602e450a8a8 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:44:38.064 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:44:38.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:44:38.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1Fypm8PFWIxAAIxbQ3oKUsROGUKueQmtIJA== --osd-uuid 5c66115a-0ff3-4f0b-bf30-b602e450a8a8 2024-04-25T08:44:38.118 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.127+0000 7f25c1d007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:38.122 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.127+0000 7f25c1d007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:38.122 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.127+0000 7f25c1d007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:38.122 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.127+0000 7f25c1d007c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:38.122 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.127+0000 7f25c1d007c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:38.122 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.127+0000 7f25c1d007c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:38.159 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:38.167+0000 7f25c1d007c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:44:44.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:44:44.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:44:44.477 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:44:44.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:44:44.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:44:45.052 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:44:45.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:44:45.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:44:45.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:44:45.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:44:45.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:44:45.085 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:45.091+0000 7f36a147f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:45.089 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:45.095+0000 7f36a147f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:45.090 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:45.099+0000 7f36a147f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:45.547 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:44:45.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:44:45.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:45.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:45.671 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:45.679+0000 7f36a147f7c0 -1 Falling back to public interface 2024-04-25T08:44:45.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:46.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:46.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:46.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:44:46.988 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:44:46.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:46.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:47.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:48.258 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:44:48.263+0000 7f36a147f7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:44:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:48.481 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:44:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:44:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:49.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:50.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:50.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:50.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:44:50.136 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:44:50.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:50.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:50.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:51.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:51.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:51.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:44:51.597 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:44:51.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:51.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:52.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:53.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:53.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:53.209 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:44:53.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:44:53.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:53.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:53.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:54.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:54.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:54.666 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:44:54.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:44:54.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:54.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:55.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:56.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:56.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:56.150 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:44:56.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:44:56.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:56.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:56.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:57.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:57.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:57.579 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:44:57.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:44:57.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:57.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:58.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:59.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:59.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:59.040 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:44:59.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:59.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:59.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:59.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:00.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:00.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:00.527 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:45:00.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:45:00.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:00.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:00.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:01.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:01.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:01.971 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:45:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:45:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:03.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:04.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:04.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:04.016 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:45:04.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:45:04.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:04.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:04.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:05.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:05.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:05.984 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:45:05.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:45:05.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:05.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:07.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:08.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:08.202 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:45:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:45:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:09.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:10.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:10.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:10.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:45:10.133 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:45:10.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:10.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:10.601 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/601755241,v1:127.0.0.1:6819/601755241] [v2:127.0.0.1:6820/601755241,v1:127.0.0.1:6821/601755241] exists,up 5c66115a-0ff3-4f0b-bf30-b602e450a8a8 2024-04-25T08:45:10.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:45:10.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:45:10.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:45:10.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bfdeb829-3373-4257-92fb-e5d3890b0ecb 2024-04-25T08:45:10.608 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 bfdeb829-3373-4257-92fb-e5d3890b0ecb 2024-04-25T08:45:10.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bfdeb829-3373-4257-92fb-e5d3890b0ecb' 2024-04-25T08:45:10.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:45:10.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWGCpmOUcZJhAAEQu3L6RNTmRnrQRyqGDi2Q== 2024-04-25T08:45:10.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWGCpmOUcZJhAAEQu3L6RNTmRnrQRyqGDi2Q=="}' 2024-04-25T08:45:10.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bfdeb829-3373-4257-92fb-e5d3890b0ecb -i td/osd-recovery-stats/3/new.json 2024-04-25T08:45:11.103 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:45:11.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:45:11.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWGCpmOUcZJhAAEQu3L6RNTmRnrQRyqGDi2Q== --osd-uuid bfdeb829-3373-4257-92fb-e5d3890b0ecb 2024-04-25T08:45:11.155 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.162+0000 7f636a4297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:11.174 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.182+0000 7f636a4297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.182+0000 7f636a4297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.182+0000 7f636a4297c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:45:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.182+0000 7f636a4297c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:45:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.182+0000 7f636a4297c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:45:11.214 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:11.222+0000 7f636a4297c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:45:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:45:16.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:45:16.996 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:45:16.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:45:16.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:45:17.583 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:45:17.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:45:17.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:45:17.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:45:17.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:45:17.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:45:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:17.622+0000 7f2ee80097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:17.631 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:17.638+0000 7f2ee80097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:17.633 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:17.638+0000 7f2ee80097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:18.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:45:18.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:45:18.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:45:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:45:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:45:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:18.057 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:45:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:45:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:18.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:18.203 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:18.210+0000 7f2ee80097c0 -1 Falling back to public interface 2024-04-25T08:45:18.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:19.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:19.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:19.541 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:45:19.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:45:19.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:19.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:20.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:20.667 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:20.674+0000 7f2ee80097c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:45:21.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:21.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:21.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:45:21.016 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:45:21.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:21.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:21.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:22.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:22.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:22.456 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:45:22.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:45:22.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:22.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:22.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:23.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:23.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:23.976 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:45:23.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:45:23.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:23.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:24.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:25.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:25.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:25.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:45:25.542 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:45:25.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:25.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:25.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:26.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:26.990 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:45:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:45:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:26.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:27.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:28.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:28.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:28.540 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:45:28.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:45:28.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:28.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:28.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:29.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:29.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:29.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:45:29.997 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:45:29.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:29.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:30.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:31.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:31.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:31.560 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:45:31.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:45:31.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:31.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:32.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:33.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:33.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:33.036 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:45:33.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:45:33.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:33.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:33.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:34.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:34.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:34.901 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:45:34.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:45:34.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:34.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:35.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:36.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:36.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:36.692 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:45:36.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:45:36.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:36.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:37.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:38.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:38.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:38.479 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:45:38.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:45:38.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:38.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:39.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:40.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:40.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:40.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:45:40.124 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:45:40.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:40.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:40.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:41.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:41.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:41.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:45:41.856 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:45:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:42.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:43.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:43.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:43.812 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:45:43.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:45:43.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:43.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:44.295 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2327768286,v1:127.0.0.1:6827/2327768286] [v2:127.0.0.1:6828/2327768286,v1:127.0.0.1:6829/2327768286] exists,up bfdeb829-3373-4257-92fb-e5d3890b0ecb 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:45:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:45:44.300 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:45:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:45:44.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:45:44.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:45:44.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:45:44.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3356b976-0fa9-46ab-85a6-ef72961876df 2024-04-25T08:45:44.302 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 3356b976-0fa9-46ab-85a6-ef72961876df 2024-04-25T08:45:44.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3356b976-0fa9-46ab-85a6-ef72961876df' 2024-04-25T08:45:44.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:45:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4GCpmwHvUExAAsREGsE8a3ooBWnpqEEHd0w== 2024-04-25T08:45:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4GCpmwHvUExAAsREGsE8a3ooBWnpqEEHd0w=="}' 2024-04-25T08:45:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3356b976-0fa9-46ab-85a6-ef72961876df -i td/osd-recovery-stats/4/new.json 2024-04-25T08:45:45.022 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:45:45.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:45:45.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4GCpmwHvUExAAsREGsE8a3ooBWnpqEEHd0w== --osd-uuid 3356b976-0fa9-46ab-85a6-ef72961876df 2024-04-25T08:45:45.078 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.086+0000 7f852d2ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:45.081 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.086+0000 7f852d2ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:45.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.090+0000 7f852d2ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:45.082 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.090+0000 7f852d2ca7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:45:45.083 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.090+0000 7f852d2ca7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:45:45.083 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.090+0000 7f852d2ca7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:45:45.144 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:45.150+0000 7f852d2ca7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:45:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:45:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:45:51.032 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T08:45:51.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:45:51.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:45:51.623 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T08:45:51.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:45:51.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:45:51.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:45:51.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:45:51.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:45:51.657 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:51.662+0000 7f361d16c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:51.659 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:51.666+0000 7f361d16c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:51.661 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:51.666+0000 7f361d16c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:52.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:45:52.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:45:52.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:45:52.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:45:52.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:45:52.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:52.074 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:45:52.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:45:52.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:52.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:52.219 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:52.226+0000 7f361d16c7c0 -1 Falling back to public interface 2024-04-25T08:45:52.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:53.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:53.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:53.541 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:45:53.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:45:53.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:53.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:53.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:54.722 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:45:54.726+0000 7f361d16c7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:45:54.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:54.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:54.973 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:45:54.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:45:54.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:54.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:55.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:56.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:56.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:56.501 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:45:56.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:45:56.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:56.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:56.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:57.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:57.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:57.974 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:45:57.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:45:57.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:57.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:58.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:59.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:59.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:59.457 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:45:59.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:45:59.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:59.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:59.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:00.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:00.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:00.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:46:00.935 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:46:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:00.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:01.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:02.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:02.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:02.492 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:46:02.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:46:02.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:02.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:02.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:03.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:03.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:03.960 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:46:03.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:46:03.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:03.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:04.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:05.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:05.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:05.509 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:46:05.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:46:05.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:05.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:06.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:07.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:07.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:07.024 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:46:07.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:07.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:07.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:07.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:08.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:08.531 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:46:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:08.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:09.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:10.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:10.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:10.229 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:46:10.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:10.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:10.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:11.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:12.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:12.053 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:46:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:13.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:14.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:14.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:14.025 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:46:14.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:14.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:14.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:14.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:15.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:15.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:15.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:15.520 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:46:15.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:15.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:16.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:17.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:17.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:17.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:17.588 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:46:17.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:17.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:18.037 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2407039498,v1:127.0.0.1:6835/2407039498] [v2:127.0.0.1:6836/2407039498,v1:127.0.0.1:6837/2407039498] exists,up 3356b976-0fa9-46ab-85a6-ef72961876df 2024-04-25T08:46:18.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:18.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:18.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:18.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:46:18.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:46:18.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:46:18.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:46:18.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:46:18.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:46:18.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:46:18.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:46:18.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:46:18.086 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:46:18.087 INFO:tasks.workunit.client.0.smithi031.stdout:add osd5 d4c78137-d348-4970-a517-8af4d3c443c2 2024-04-25T08:46:18.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4c78137-d348-4970-a517-8af4d3c443c2 2024-04-25T08:46:18.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 d4c78137-d348-4970-a517-8af4d3c443c2' 2024-04-25T08:46:18.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:46:18.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaGCpmmm4lBxAAP5Cn2C2IMKTOssCnZvUm5w== 2024-04-25T08:46:18.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaGCpmmm4lBxAAP5Cn2C2IMKTOssCnZvUm5w=="}' 2024-04-25T08:46:18.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4c78137-d348-4970-a517-8af4d3c443c2 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:46:18.721 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:46:18.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:46:18.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaGCpmmm4lBxAAP5Cn2C2IMKTOssCnZvUm5w== --osd-uuid d4c78137-d348-4970-a517-8af4d3c443c2 2024-04-25T08:46:18.769 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.773+0000 7f47ebed77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:18.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.789+0000 7f47ebed77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:18.783 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.789+0000 7f47ebed77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:18.783 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.789+0000 7f47ebed77c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:46:18.784 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.789+0000 7f47ebed77c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:46:18.784 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.789+0000 7f47ebed77c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:46:18.822 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:18.829+0000 7f47ebed77c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:46:24.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:46:24.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:46:24.952 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd5 key to auth repository 2024-04-25T08:46:24.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:46:24.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:46:25.540 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.5 2024-04-25T08:46:25.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:46:25.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:46:25.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:46:25.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:46:25.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:46:25.573 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:25.577+0000 7f8eacd7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:25.575 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:25.581+0000 7f8eacd7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:25.576 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:25.581+0000 7f8eacd7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:26.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:46:26.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:46:26.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:46:26.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:46:26.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:46:26.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:26.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:46:26.045 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:46:26.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:26.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:26.119 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:26.125+0000 7f8eacd7d7c0 -1 Falling back to public interface 2024-04-25T08:46:26.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:27.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:27.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:27.485 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:46:27.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:46:27.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:27.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:27.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:28.637 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:46:28.641+0000 7f8eacd7d7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:46:28.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:28.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:28.998 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:46:28.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:46:28.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:28.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:29.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:30.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:30.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:30.443 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:46:30.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:46:30.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:30.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:30.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:31.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:31.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:31.935 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:46:31.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:46:31.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:31.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:32.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:46:33.426 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:46:33.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:33.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:34.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:35.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:35.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:35.016 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:46:35.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:46:35.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:35.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:35.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:36.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:36.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:36.545 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:46:36.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:46:36.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:36.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:36.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:37.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:37.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:37.981 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:46:37.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:46:37.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:37.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:38.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:39.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:39.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:39.431 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:46:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:46:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:39.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:39.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:40.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:40.995 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:46:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:41.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:42.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:42.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:42.767 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:46:42.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:42.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:42.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:43.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:44.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:44.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:44.669 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:46:44.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:44.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:44.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:45.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:46.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:46.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:46.427 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:46:46.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:46.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:46.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:47.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:48.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:48.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:48.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:48.135 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:46:48.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:48.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:49.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:50.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:50.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:50.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:50.015 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:46:50.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:50.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:50.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:51.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:51.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:51.462 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:46:51.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:51.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:51.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:51.966 INFO:tasks.workunit.client.0.smithi031.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/694843208,v1:127.0.0.1:6843/694843208] [v2:127.0.0.1:6844/694843208,v1:127.0.0.1:6845/694843208] exists,up d4c78137-d348-4970-a517-8af4d3c443c2 2024-04-25T08:46:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-04-25T08:46:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:46:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:46:52.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:46:53.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-04-25T08:46:54.465 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 4 2024-04-25T08:46:54.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:46:54.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:46:54.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:46:54.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:46:54.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:46:54.492 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:46:54.492 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:46:54.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:46:54.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:46:54.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:46:54.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:46:54.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:46:54.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:46:54.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:46:54.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:46:54.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:46:55.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:46:55.066 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:46:55.066 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:46:55.067 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:46:55.067 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:46:55.067 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:46:55.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:46:55.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:55.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:46:55.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836523 2024-04-25T08:46:55.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836523 2024-04-25T08:46:55.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523' 2024-04-25T08:46:55.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:55.234 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:46:55.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672996 2024-04-25T08:46:55.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672996 2024-04-25T08:46:55.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672996' 2024-04-25T08:46:55.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:55.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:46:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509468 2024-04-25T08:46:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509468 2024-04-25T08:46:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672996 2-64424509468' 2024-04-25T08:46:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:55.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:46:55.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T08:46:55.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T08:46:55.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672996 2-64424509468 3-85899345941' 2024-04-25T08:46:55.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:55.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:46:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182414 2024-04-25T08:46:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182414 2024-04-25T08:46:55.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672996 2-64424509468 3-85899345941 4-107374182414' 2024-04-25T08:46:55.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:55.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:46:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018889 2024-04-25T08:46:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018889 2024-04-25T08:46:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672996 2-64424509468 3-85899345941 4-107374182414 5-128849018889' 2024-04-25T08:46:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:56.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836523 2024-04-25T08:46:56.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:56.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:46:56.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836523 2024-04-25T08:46:56.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:56.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836523 2024-04-25T08:46:56.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-04-25T08:46:56.034 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836523 2024-04-25T08:46:56.034 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:46:56.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836523 -lt 21474836523 2024-04-25T08:46:56.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:56.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672996 2024-04-25T08:46:56.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:56.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:46:56.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:56.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672996 2024-04-25T08:46:56.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672996 2024-04-25T08:46:56.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-04-25T08:46:56.453 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672996 2024-04-25T08:46:56.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:46:56.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672994 -lt 42949672996 2024-04-25T08:46:56.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:46:57.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:46:57.940 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:46:58.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672997 -lt 42949672996 2024-04-25T08:46:58.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:58.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:58.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509468 2024-04-25T08:46:58.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:46:58.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509468 2024-04-25T08:46:58.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:58.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509468 2024-04-25T08:46:58.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-04-25T08:46:58.428 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509468 2024-04-25T08:46:58.428 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:46:58.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509468 2024-04-25T08:46:58.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:58.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345941 2024-04-25T08:46:58.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:58.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:46:58.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-04-25T08:46:58.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:58.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T08:46:58.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-04-25T08:46:58.872 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345941 2024-04-25T08:46:58.873 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:46:59.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345941 2024-04-25T08:46:59.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:59.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182414 2024-04-25T08:46:59.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:59.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:46:59.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182414 2024-04-25T08:46:59.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:59.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182414 2024-04-25T08:46:59.319 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182414 2024-04-25T08:46:59.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-04-25T08:46:59.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:46:59.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182414 2024-04-25T08:46:59.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:59.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018889 2024-04-25T08:46:59.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:59.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:46:59.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018889 2024-04-25T08:46:59.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:59.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018889 2024-04-25T08:46:59.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2024-04-25T08:46:59.809 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018889 2024-04-25T08:46:59.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:47:00.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018890 -lt 128849018889 2024-04-25T08:47:00.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:47:00.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:00.229 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:00.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:00.837 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:01.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:47:01.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:01.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:01.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:01.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:47:01.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:47:01.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:47:01.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-04-25T08:47:01.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:01.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-04-25T08:47:02.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-04-25T08:47:02.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-04-25T08:47:02.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-04-25T08:47:02.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-04-25T08:47:02.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-04-25T08:47:02.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-04-25T08:47:02.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:02.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-04-25T08:47:03.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-04-25T08:47:03.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-04-25T08:47:03.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-04-25T08:47:03.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-04-25T08:47:03.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-04-25T08:47:03.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-04-25T08:47:03.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-04-25T08:47:03.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:03.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-04-25T08:47:04.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-04-25T08:47:04.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-04-25T08:47:04.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-04-25T08:47:04.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-04-25T08:47:04.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-04-25T08:47:04.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-04-25T08:47:04.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-04-25T08:47:04.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:04.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-04-25T08:47:05.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:05.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-04-25T08:47:05.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:05.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-04-25T08:47:05.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:05.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-04-25T08:47:05.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:05.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-04-25T08:47:05.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:05.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-04-25T08:47:05.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:05.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-04-25T08:47:06.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-04-25T08:47:06.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-04-25T08:47:06.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-04-25T08:47:06.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-04-25T08:47:06.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-04-25T08:47:06.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-04-25T08:47:06.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-04-25T08:47:06.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:06.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-04-25T08:47:07.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-04-25T08:47:07.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-04-25T08:47:07.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-04-25T08:47:07.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-04-25T08:47:07.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-04-25T08:47:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-04-25T08:47:07.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:07.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-04-25T08:47:08.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-04-25T08:47:08.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-04-25T08:47:08.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-04-25T08:47:08.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-04-25T08:47:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-04-25T08:47:08.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-04-25T08:47:08.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-04-25T08:47:08.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:08.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-04-25T08:47:09.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-04-25T08:47:09.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-04-25T08:47:09.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-04-25T08:47:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-04-25T08:47:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-04-25T08:47:09.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-04-25T08:47:09.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:09.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-04-25T08:47:10.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-04-25T08:47:10.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-04-25T08:47:10.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-04-25T08:47:10.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-04-25T08:47:10.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-04-25T08:47:10.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-04-25T08:47:10.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-04-25T08:47:10.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:10.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-04-25T08:47:11.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-04-25T08:47:11.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-04-25T08:47:11.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-04-25T08:47:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-04-25T08:47:11.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-04-25T08:47:11.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-04-25T08:47:11.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:11.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-04-25T08:47:12.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-04-25T08:47:12.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-04-25T08:47:12.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-04-25T08:47:12.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-04-25T08:47:12.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-04-25T08:47:12.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-04-25T08:47:12.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-04-25T08:47:12.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:12.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-04-25T08:47:13.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-04-25T08:47:13.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-04-25T08:47:13.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-04-25T08:47:13.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-04-25T08:47:13.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-04-25T08:47:13.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-04-25T08:47:13.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:13.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-04-25T08:47:14.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-04-25T08:47:14.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-04-25T08:47:14.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-04-25T08:47:14.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-04-25T08:47:14.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-04-25T08:47:14.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-04-25T08:47:14.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-04-25T08:47:14.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:14.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-04-25T08:47:15.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-04-25T08:47:15.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-04-25T08:47:15.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-04-25T08:47:15.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-04-25T08:47:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-04-25T08:47:15.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-04-25T08:47:15.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:15.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-04-25T08:47:16.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-04-25T08:47:16.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-04-25T08:47:16.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-04-25T08:47:16.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-04-25T08:47:16.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-04-25T08:47:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-04-25T08:47:16.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:16.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-04-25T08:47:17.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-04-25T08:47:17.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-04-25T08:47:17.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-04-25T08:47:17.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-04-25T08:47:17.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-04-25T08:47:17.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-04-25T08:47:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:17.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-04-25T08:47:18.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-04-25T08:47:18.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-04-25T08:47:18.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-04-25T08:47:18.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-04-25T08:47:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-04-25T08:47:18.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-04-25T08:47:18.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-04-25T08:47:18.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:18.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-04-25T08:47:19.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-04-25T08:47:19.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-04-25T08:47:19.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-04-25T08:47:19.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-04-25T08:47:19.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-04-25T08:47:19.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-04-25T08:47:19.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-04-25T08:47:19.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:19.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-04-25T08:47:20.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-04-25T08:47:20.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-04-25T08:47:20.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-04-25T08:47:20.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-04-25T08:47:20.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-04-25T08:47:20.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-04-25T08:47:20.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:20.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-04-25T08:47:21.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-04-25T08:47:21.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-04-25T08:47:21.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-04-25T08:47:21.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-04-25T08:47:21.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-04-25T08:47:21.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-04-25T08:47:21.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:21.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-04-25T08:47:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-04-25T08:47:22.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-04-25T08:47:22.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-04-25T08:47:22.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-04-25T08:47:22.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-04-25T08:47:22.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-04-25T08:47:22.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-04-25T08:47:22.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:22.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-04-25T08:47:23.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-04-25T08:47:23.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-04-25T08:47:23.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-04-25T08:47:23.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-04-25T08:47:23.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-04-25T08:47:23.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-04-25T08:47:23.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:23.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-04-25T08:47:24.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-04-25T08:47:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-04-25T08:47:24.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-04-25T08:47:24.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-04-25T08:47:24.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-04-25T08:47:24.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-04-25T08:47:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-04-25T08:47:24.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:24.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-04-25T08:47:25.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-04-25T08:47:25.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-04-25T08:47:25.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-04-25T08:47:25.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-04-25T08:47:25.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-04-25T08:47:25.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-04-25T08:47:25.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:25.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-04-25T08:47:26.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-04-25T08:47:26.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-04-25T08:47:26.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-04-25T08:47:26.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-04-25T08:47:26.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-04-25T08:47:26.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-04-25T08:47:26.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:26.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-04-25T08:47:27.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-04-25T08:47:27.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-04-25T08:47:27.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-04-25T08:47:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-04-25T08:47:27.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-04-25T08:47:27.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-04-25T08:47:27.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-04-25T08:47:27.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:27.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-04-25T08:47:28.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:28.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-04-25T08:47:28.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:28.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-04-25T08:47:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-04-25T08:47:28.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:28.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-04-25T08:47:28.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:28.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-04-25T08:47:28.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:47:28.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-04-25T08:47:28.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:47:28.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:47:28.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:47:28.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:47:28.921 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:47:29.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-04-25T08:47:29.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-04-25T08:47:29.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:47:29.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:47:29.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:47:29.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:47:29.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-04-25T08:47:29.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-04-25T08:47:29.828 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T08:47:29.828 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-04-25T08:47:29.828 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:47:29.828 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr:0 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr:4' 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-04-25T08:47:30.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-04-25T08:47:30.987 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:47:31.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:47:31.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-04-25T08:47:31.895 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already out. 2024-04-25T08:47:31.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:47:31.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-04-25T08:47:32.794 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T08:47:32.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:47:32.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-04-25T08:47:34.061 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already out. 2024-04-25T08:47:34.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:47:34.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-04-25T08:47:35.136 INFO:tasks.workunit.client.0.smithi031.stderr:osd.4 is already out. 2024-04-25T08:47:35.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-04-25T08:47:36.500 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T08:47:36.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-04-25T08:47:38.217 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:47:38.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:47:38.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:47:38.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:47:38.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:47:38.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:47:38.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-04-25T08:47:38.848 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:47:38.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-04-25T08:47:40.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:47:40.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:47:40.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:47:40.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:47:40.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:47:40.864 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:47:40.864 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:47:40.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:47:40.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:47:40.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:47:40.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:47:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:47:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:47:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:47:40.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:47:40.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:47:41.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:47:41.436 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:47:41.436 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:47:41.437 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:47:41.437 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:47:41.437 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:47:41.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:47:41.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.437 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:47:41.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836534 2024-04-25T08:47:41.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836534 2024-04-25T08:47:41.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534' 2024-04-25T08:47:41.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.598 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:47:41.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673008 2024-04-25T08:47:41.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673008 2024-04-25T08:47:41.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949673008' 2024-04-25T08:47:41.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:47:42.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509480 2024-04-25T08:47:42.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509480 2024-04-25T08:47:42.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509480' 2024-04-25T08:47:42.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:42.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:47:42.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345952 2024-04-25T08:47:42.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345952 2024-04-25T08:47:42.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509480 3-85899345952' 2024-04-25T08:47:42.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:42.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:47:42.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182425 2024-04-25T08:47:42.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182425 2024-04-25T08:47:42.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509480 3-85899345952 4-107374182425' 2024-04-25T08:47:42.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:42.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:47:42.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018901 2024-04-25T08:47:42.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018901 2024-04-25T08:47:42.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509480 3-85899345952 4-107374182425 5-128849018901' 2024-04-25T08:47:42.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:42.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836534 2024-04-25T08:47:42.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:42.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:47:42.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836534 2024-04-25T08:47:42.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:42.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836534 2024-04-25T08:47:42.544 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836534 2024-04-25T08:47:42.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-04-25T08:47:42.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:47:42.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836532 -lt 21474836534 2024-04-25T08:47:42.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:47:43.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:47:43.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:47:44.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836535 -lt 21474836534 2024-04-25T08:47:44.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:44.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673008 2024-04-25T08:47:44.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:44.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:47:44.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673008 2024-04-25T08:47:44.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:44.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673008 2024-04-25T08:47:44.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-04-25T08:47:44.410 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673008 2024-04-25T08:47:44.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:47:44.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673008 -lt 42949673008 2024-04-25T08:47:44.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:44.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509480 2024-04-25T08:47:44.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:44.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:47:44.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509480 2024-04-25T08:47:44.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:44.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509480 2024-04-25T08:47:44.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2024-04-25T08:47:44.887 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509480 2024-04-25T08:47:44.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:47:45.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509480 -lt 64424509480 2024-04-25T08:47:45.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:45.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345952 2024-04-25T08:47:45.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:45.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:47:45.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345952 2024-04-25T08:47:45.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:45.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345952 2024-04-25T08:47:45.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2024-04-25T08:47:45.393 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345952 2024-04-25T08:47:45.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:47:46.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345953 -lt 85899345952 2024-04-25T08:47:46.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:46.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182425 2024-04-25T08:47:46.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:46.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:47:46.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182425 2024-04-25T08:47:46.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:46.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182425 2024-04-25T08:47:46.011 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182425 2024-04-25T08:47:46.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-04-25T08:47:46.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:47:46.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182426 -lt 107374182425 2024-04-25T08:47:46.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:46.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018901 2024-04-25T08:47:46.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:46.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:47:46.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018901 2024-04-25T08:47:46.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:46.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018901 2024-04-25T08:47:46.504 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018901 2024-04-25T08:47:46.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-04-25T08:47:46.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:47:46.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018901 -lt 128849018901 2024-04-25T08:47:47.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:47:47.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:47.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:47.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:47:47.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:47.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:47.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:47.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:47.700 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:47.701 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:47.701 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:48.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:48.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:48.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:48.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:48.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:48.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:47:48.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:47:48.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:47:48.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:48.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:48.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:48.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:48.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:48.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:48.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:48.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:48.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:48.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:49.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:49.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:49.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:49.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:50.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:50.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:50.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:50.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:47:50.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:47:50.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:50.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:50.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:50.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:50.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:50.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:50.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:50.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:50.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:50.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:50.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:51.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:51.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:51.322 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:51.323 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:51.890 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:52.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:47:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:47:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:52.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:52.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:52.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:52.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:52.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:52.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:52.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:52.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:53.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:53.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:53.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:53.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:53.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:53.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:53.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:53.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:53.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:53.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:53.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:53.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:53.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:47:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:47:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:54.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:54.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:54.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:54.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:54.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:54.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:54.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:54.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:54.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:55.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:55.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:55.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:55.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:55.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:55.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:55.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:56.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:47:56.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:47:56.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:56.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:56.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:56.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:56.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:56.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:56.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:56.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:56.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:56.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:56.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:56.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:56.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:56.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:56.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:57.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:57.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:57.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:58.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:47:58.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:47:58.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:58.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:58.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:58.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:58.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:58.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:58.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:58.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:58.231 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:58.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:58.232 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:58.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:58.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:58.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:58.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:59.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:59.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:59.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:59.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:47:59.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:47:59.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:59.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:59.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:59.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:59.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:59.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:59.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:59.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:59.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:59.942 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:59.942 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:00.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:48:00.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:00.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:00.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:00.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:48:00.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:48:00.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:48:00.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:48:00.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:48:00.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:48:00.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:48:00.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:48:01.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-04-25T08:48:01.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-04-25T08:48:01.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.321 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:48:01.322 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:48:01.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-04-25T08:48:01.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:48:01.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:48:01.338 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.338 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-04-25T08:48:01.338 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:48:01.338 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:48:01.338 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:48:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:48:01.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:48:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:48:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:48:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:48:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:48:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:48:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:48:01.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:48:01.352 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.352 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-04-25T08:48:01.352 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:48:01.352 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:48:01.352 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:48:01.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.355 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:48:01.356 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:48:01.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:48:01.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-04-25T08:48:01.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:48:01.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-04-25T08:48:01.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-04-25T08:48:01.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-04-25T08:48:01.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:48:01.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-04-25T08:48:01.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:48:02.010 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:48:02.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-04-25T08:48:02.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:48:02.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:48:02.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:48:02.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:48:02.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:48:07.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:48:07.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:48:07.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:48:07.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:48:07.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:48:07.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:48:07.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:48:07.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:48:07.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:48:07.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:48:07.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:48:07.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:48:07.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:48:07.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:48:07.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:48:07.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:48:07.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:48:07.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:07.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:48:07.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:48:07.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:07.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:48:07.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:48:07.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:48:07.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:48:07.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:07.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:07.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:48:07.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:48:07.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:48:07.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:48:07.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:48:07.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:48:07.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:48:07.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:48:07.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:48:07.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:48:07.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:48:07.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:48:07.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:48:07.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:07.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:48:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:48:07.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:07.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:48:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:48:07.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:48:07.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:48:07.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:07.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:07.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:48:07.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:48:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:48:07.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:48:07.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:48:07.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:07.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:07.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:48:07.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:48:07.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:48:07.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:48:07.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:07.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:07.894 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:07.894 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:07.894 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:07.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:48:07.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:48:08.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:48:08.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:48:08.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:48:08.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:48:08.131 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:48:08.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:48:08.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:48:08.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:48:08.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:48:08.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:48:08.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:48:08.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:48:08.255 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:48:08.255 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:48:08.255 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:48:08.255 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:48:08.255 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:08.256 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:08.256 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:48:08.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:48:08.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:48:08.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-04-25T08:48:08.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:48:08.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:48:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:48:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:48:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:48:08.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:48:09.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:48:09.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:09.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:09.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:09.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:09.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:09.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:48:09.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:48:09.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:48:09.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-04-25T08:48:09.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:48:09.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:09.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:48:09.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8bf56270-c7a1-4de5-b7f0-8a3bb76778d3 2024-04-25T08:48:09.366 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 8bf56270-c7a1-4de5-b7f0-8a3bb76778d3 2024-04-25T08:48:09.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8bf56270-c7a1-4de5-b7f0-8a3bb76778d3' 2024-04-25T08:48:09.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:48:09.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJGCpmNBVxFxAAOC6EO6zF/aEwtnVL96mqbQ== 2024-04-25T08:48:09.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJGCpmNBVxFxAAOC6EO6zF/aEwtnVL96mqbQ=="}' 2024-04-25T08:48:09.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8bf56270-c7a1-4de5-b7f0-8a3bb76778d3 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:48:09.674 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:48:09.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:48:09.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJGCpmNBVxFxAAOC6EO6zF/aEwtnVL96mqbQ== --osd-uuid 8bf56270-c7a1-4de5-b7f0-8a3bb76778d3 2024-04-25T08:48:09.719 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.727+0000 7fee5e2567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:09.735 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.743+0000 7fee5e2567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:09.736 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.743+0000 7fee5e2567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:09.736 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.743+0000 7fee5e2567c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:48:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.743+0000 7fee5e2567c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:48:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.743+0000 7fee5e2567c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:48:09.775 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:09.783+0000 7fee5e2567c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:48:15.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:48:15.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:48:15.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:48:15.494 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:48:15.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:48:16.066 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:48:16.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:48:16.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:48:16.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:48:16.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:48:16.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:48:16.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:16.107+0000 7f362daa67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:16.132 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:16.139+0000 7f362daa67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:16.133 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:16.143+0000 7f362daa67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:16.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:48:16.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:48:16.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:48:16.567 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:48:16.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:48:16.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:48:16.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:16.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:48:16.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:16.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:16.690 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:16.699+0000 7f362daa67c0 -1 Falling back to public interface 2024-04-25T08:48:17.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:18.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:18.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:18.047 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:48:18.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:48:18.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:18.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:18.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:19.036 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:19.043+0000 7f362daa67c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:48:19.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:19.505 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:48:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:48:19.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:19.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:20.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:21.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:21.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:21.091 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:48:21.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:48:21.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:21.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:21.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:22.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:22.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:22.533 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:48:22.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:48:22.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:22.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:23.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:24.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:24.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:24.100 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:48:24.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:48:24.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:24.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:24.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:25.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:25.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:25.557 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:48:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:48:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:25.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:26.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:27.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:27.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:27.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:48:27.103 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:48:27.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:27.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:27.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:28.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:28.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:28.551 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:48:28.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:48:28.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:28.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:29.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:30.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:30.066 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:48:30.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:48:30.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:30.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:30.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:31.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:31.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:31.703 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:48:31.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:48:31.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:31.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:32.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:33.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:33.782 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:48:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:48:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:34.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:35.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:35.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:35.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:48:35.737 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:48:35.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:35.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:36.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:37.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:48:37.472 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:48:37.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:37.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:38.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:39.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:39.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:39.869 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:48:39.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:48:39.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:39.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:40.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:41.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:41.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:41.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:48:41.597 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:48:41.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:41.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:42.062 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3368845697,v1:127.0.0.1:6803/3368845697] [v2:127.0.0.1:6804/3368845697,v1:127.0.0.1:6805/3368845697] exists,up 8bf56270-c7a1-4de5-b7f0-8a3bb76778d3 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:48:42.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:48:42.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:42.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:42.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:42.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:42.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:48:42.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:48:42.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:48:42.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:48:42.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8208d91-b672-4492-a643-281495a63286 2024-04-25T08:48:42.070 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 d8208d91-b672-4492-a643-281495a63286 2024-04-25T08:48:42.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d8208d91-b672-4492-a643-281495a63286' 2024-04-25T08:48:42.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:48:42.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqGCpmREYeBhAA9Z+76BasINbeaoS5z4ZgDg== 2024-04-25T08:48:42.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqGCpmREYeBhAA9Z+76BasINbeaoS5z4ZgDg=="}' 2024-04-25T08:48:42.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8208d91-b672-4492-a643-281495a63286 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:48:42.677 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:48:42.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:48:42.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqGCpmREYeBhAA9Z+76BasINbeaoS5z4ZgDg== --osd-uuid d8208d91-b672-4492-a643-281495a63286 2024-04-25T08:48:42.731 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.739+0000 7f0beb9f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:42.763 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.771+0000 7f0beb9f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:42.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.771+0000 7f0beb9f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:42.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.771+0000 7f0beb9f37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:48:42.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.771+0000 7f0beb9f37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:48:42.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.771+0000 7f0beb9f37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:48:42.802 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:42.811+0000 7f0beb9f37c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:48:48.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:48:48.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:48:48.803 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:48:48.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:48:48.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:48:49.381 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:48:49.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:48:49.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:48:49.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:48:49.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:48:49.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:48:49.415 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:49.423+0000 7f80be9f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:49.433 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:49.443+0000 7f80be9f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:49.434 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:49.443+0000 7f80be9f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:49.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:48:49.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:48:49.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:48:49.804 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:48:49.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:48:49.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:48:49.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:49.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:48:49.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:49.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:49.992 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:49.999+0000 7f80be9f57c0 -1 Falling back to public interface 2024-04-25T08:48:50.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:51.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:51.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:51.243 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:48:51.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:48:51.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:51.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:51.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:52.609 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:48:52.615+0000 7f80be9f57c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:48:52.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:52.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:52.671 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:48:52.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:48:52.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:52.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:53.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:54.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:54.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:54.277 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:48:54.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:48:54.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:54.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:54.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:55.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:55.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:55.764 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:48:55.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:48:55.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:55.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:56.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:57.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:57.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:57.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:48:57.234 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:48:57.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:57.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:57.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:58.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:58.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:58.694 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:48:58.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:48:58.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:58.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:59.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:00.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:00.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:00.219 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:49:00.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:49:00.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:00.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:00.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:01.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:01.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:01.874 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:49:01.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:49:01.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:01.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:02.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:03.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:03.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:03.457 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:49:03.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:49:03.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:03.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:03.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:04.886 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:49:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:49:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:05.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:06.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:06.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:06.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:49:06.736 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:49:06.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:06.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:07.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:08.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:08.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:08.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:49:08.656 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:49:08.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:08.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:09.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:10.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:10.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:10.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:49:10.822 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:49:10.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:10.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:11.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:12.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:12.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:12.849 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:49:12.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:49:12.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:12.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:13.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:14.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:14.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:14.887 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:49:14.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:49:14.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:14.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:15.384 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/404455685,v1:127.0.0.1:6811/404455685] [v2:127.0.0.1:6812/404455685,v1:127.0.0.1:6813/404455685] exists,up d8208d91-b672-4492-a643-281495a63286 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:49:15.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:49:15.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:49:15.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:49:15.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:49:15.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:49:15.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94222062-e7ed-47ec-b699-37c3c1b7938b 2024-04-25T08:49:15.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 94222062-e7ed-47ec-b699-37c3c1b7938b' 2024-04-25T08:49:15.392 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 94222062-e7ed-47ec-b699-37c3c1b7938b 2024-04-25T08:49:15.392 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:49:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALGSpmb4xQGRAAGRK8+jQobCOmCVZDR0ds0g== 2024-04-25T08:49:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALGSpmb4xQGRAAGRK8+jQobCOmCVZDR0ds0g=="}' 2024-04-25T08:49:15.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94222062-e7ed-47ec-b699-37c3c1b7938b -i td/osd-recovery-stats/2/new.json 2024-04-25T08:49:16.042 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:49:16.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:49:16.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALGSpmb4xQGRAAGRK8+jQobCOmCVZDR0ds0g== --osd-uuid 94222062-e7ed-47ec-b699-37c3c1b7938b 2024-04-25T08:49:16.105 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.110+0000 7f752c0517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:16.107 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.114+0000 7f752c0517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:16.109 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.114+0000 7f752c0517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:16.109 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.118+0000 7f752c0517c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:49:16.109 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.118+0000 7f752c0517c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:49:16.109 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.118+0000 7f752c0517c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:49:16.198 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:16.206+0000 7f752c0517c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:49:22.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:49:22.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:49:22.252 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:49:22.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:49:22.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:49:22.821 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:49:22.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:49:22.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:49:22.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:49:22.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:49:22.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:49:22.853 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:22.862+0000 7ffb56d047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:22.873 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:22.878+0000 7ffb56d047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:22.874 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:22.882+0000 7ffb56d047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:49:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:49:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:49:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:49:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:49:23.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:23.269 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:49:23.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:49:23.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:23.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:23.439 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:23.446+0000 7ffb56d047c0 -1 Falling back to public interface 2024-04-25T08:49:23.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:24.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:24.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:24.703 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:49:24.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:49:24.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:24.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:25.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:26.006 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:26.014+0000 7ffb56d047c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:49:26.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:26.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:26.138 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:49:26.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:49:26.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:26.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:26.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:27.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:27.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:27.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:49:27.720 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:49:27.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:27.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:28.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:29.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:29.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:29.168 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:49:29.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:49:29.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:29.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:29.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:30.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:30.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:30.594 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:49:30.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:49:30.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:30.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:31.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:32.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:32.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:32.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:49:32.112 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:49:32.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:32.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:32.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:33.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:33.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:33.535 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:49:33.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:49:33.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:33.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:34.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:35.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:35.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:35.090 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:49:35.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:49:35.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:35.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:35.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:36.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:36.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:36.520 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:49:36.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:49:36.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:36.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:37.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:38.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:38.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:38.108 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:49:38.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:49:38.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:38.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:39.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:40.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:40.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:40.074 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:49:40.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:49:40.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:40.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:41.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:42.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:42.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:42.466 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:49:42.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:49:42.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:42.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:43.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:44.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:44.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:44.183 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:49:44.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:49:44.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:44.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:45.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:46.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:46.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:46.589 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:49:46.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:49:46.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:46.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:47.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:48.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:48.526 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:49:48.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:49:48.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:48.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:48.960 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/448816114,v1:127.0.0.1:6819/448816114] [v2:127.0.0.1:6820/448816114,v1:127.0.0.1:6821/448816114] exists,up 94222062-e7ed-47ec-b699-37c3c1b7938b 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:49:48.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:49:48.964 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:49:48.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:49:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:49:48.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:49:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ea5ed90-3afc-42ce-8bc2-9bf825004533 2024-04-25T08:49:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9ea5ed90-3afc-42ce-8bc2-9bf825004533' 2024-04-25T08:49:48.967 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 9ea5ed90-3afc-42ce-8bc2-9bf825004533 2024-04-25T08:49:48.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:49:48.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsGSpmxIJ+OxAAhYQGVjIbgn2GYgyO0GZWdQ== 2024-04-25T08:49:48.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsGSpmxIJ+OxAAhYQGVjIbgn2GYgyO0GZWdQ=="}' 2024-04-25T08:49:48.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ea5ed90-3afc-42ce-8bc2-9bf825004533 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:49:49.597 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:49:49.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:49:49.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsGSpmxIJ+OxAAhYQGVjIbgn2GYgyO0GZWdQ== --osd-uuid 9ea5ed90-3afc-42ce-8bc2-9bf825004533 2024-04-25T08:49:49.644 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.650+0000 7f86a68ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:49.653 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.662+0000 7f86a68ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:49.654 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.662+0000 7f86a68ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:49.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.662+0000 7f86a68ef7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:49:49.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.662+0000 7f86a68ef7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:49:49.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.662+0000 7f86a68ef7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:49:49.726 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:49.734+0000 7f86a68ef7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:49:55.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:49:55.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:49:55.879 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:49:56.455 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:49:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:49:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:49:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:49:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:49:56.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:49:56.489 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:56.494+0000 7fd55d7237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:56.502 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:56.510+0000 7fd55d7237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:56.503 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:56.510+0000 7fd55d7237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:56.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:49:56.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:49:56.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:49:56.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:49:56.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:49:56.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:56.882 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:49:56.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:49:56.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:56.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:49:57.062 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:57.070+0000 7fd55d7237c0 -1 Falling back to public interface 2024-04-25T08:49:57.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:58.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:58.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:58.350 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:49:58.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:49:58.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:58.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:49:58.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:59.569 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:49:59.574+0000 7fd55d7237c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:49:59.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:59.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:59.811 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:49:59.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:49:59.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:59.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:00.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:01.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:01.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:01.431 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:50:01.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:50:01.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:01.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:01.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:02.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:02.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:02.879 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:50:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:50:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:02.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:03.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:04.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:04.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:04.393 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:50:04.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:50:04.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:04.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:04.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:05.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:05.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:05.889 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:50:05.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:50:05.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:05.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:06.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:07.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:07.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:07.542 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:50:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:50:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:07.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:07.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:08.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:08.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:08.993 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:50:08.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:50:08.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:08.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:09.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:10.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:10.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:10.504 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:50:10.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:50:10.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:10.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:11.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:12.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:12.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:12.462 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:50:12.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:50:12.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:12.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:13.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:14.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:14.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:14.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:50:14.361 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:50:14.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:14.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:15.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:16.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:16.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:16.716 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:50:16.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:50:16.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:16.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:18.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:18.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:18.747 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:50:18.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:50:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:19.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:20.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:20.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:20.228 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:50:20.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:50:20.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:20.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:21.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:22.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:22.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:22.423 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:50:22.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:50:22.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:22.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:22.856 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2172459031,v1:127.0.0.1:6827/2172459031] [v2:127.0.0.1:6828/2172459031,v1:127.0.0.1:6829/2172459031] exists,up 9ea5ed90-3afc-42ce-8bc2-9bf825004533 2024-04-25T08:50:22.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:50:22.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:50:22.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:50:22.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:50:22.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:50:22.863 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 51643d85-1c66-4693-bb7c-6a84ba6412e3 2024-04-25T08:50:22.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51643d85-1c66-4693-bb7c-6a84ba6412e3 2024-04-25T08:50:22.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 51643d85-1c66-4693-bb7c-6a84ba6412e3' 2024-04-25T08:50:22.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:50:22.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOGSpmbPNgNRAA92okYo1rG1AJzzW5dsH/MA== 2024-04-25T08:50:22.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOGSpmbPNgNRAA92okYo1rG1AJzzW5dsH/MA=="}' 2024-04-25T08:50:22.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51643d85-1c66-4693-bb7c-6a84ba6412e3 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:50:23.397 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:50:23.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:50:23.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOGSpmbPNgNRAA92okYo1rG1AJzzW5dsH/MA== --osd-uuid 51643d85-1c66-4693-bb7c-6a84ba6412e3 2024-04-25T08:50:23.449 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.457+0000 7fef9a9db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:23.469 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.477+0000 7fef9a9db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:23.471 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.477+0000 7fef9a9db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:23.472 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.477+0000 7fef9a9db7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:50:23.472 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.477+0000 7fef9a9db7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:50:23.472 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.477+0000 7fef9a9db7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:50:23.545 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:23.549+0000 7fef9a9db7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:50:29.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:50:29.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:50:29.648 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T08:50:29.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:50:29.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:50:30.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:50:30.228 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T08:50:30.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:50:30.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:50:30.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:50:30.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:50:30.261 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:30.265+0000 7f8de70b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:30.268 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:30.273+0000 7f8de70b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:30.269 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:30.277+0000 7f8de70b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:30.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:50:30.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:50:30.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:50:30.686 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:50:30.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:50:30.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:50:30.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:30.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:50:30.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:30.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:30.842 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:30.849+0000 7f8de70b37c0 -1 Falling back to public interface 2024-04-25T08:50:31.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:32.160 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:50:32.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:50:32.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:32.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:32.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:33.354 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:33.361+0000 7f8de70b37c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:50:33.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:33.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:33.593 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:50:33.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:50:33.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:33.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:34.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:35.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:35.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:35.144 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:50:35.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:50:35.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:35.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:35.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:36.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:36.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:36.617 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:50:36.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:50:36.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:36.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:37.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:38.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:38.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:38.135 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:50:38.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:50:38.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:38.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:38.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:39.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:39.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:39.659 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:50:39.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:50:39.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:39.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:41.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:41.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:41.185 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:50:41.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:50:41.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:41.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:41.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:42.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:42.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:42.625 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:50:42.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:50:42.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:42.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:43.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:44.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:44.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:44.138 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:50:44.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:50:44.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:44.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:44.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:45.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:45.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:45.679 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:50:45.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:50:45.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:45.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:46.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:47.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:47.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:47.284 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:50:47.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:50:47.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:47.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:48.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:49.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:49.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:49.058 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:50:49.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:50:49.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:49.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:49.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:50.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:50.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:50.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:50:50.908 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:50:50.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:50.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:52.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:52.937 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:50:52.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:52.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:50:52.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:52.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:54.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:55.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:55.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:55.037 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:50:55.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:50:55.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:55.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:55.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:56.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:56.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:56.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:50:56.734 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:50:56.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:56.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:57.175 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3958442887,v1:127.0.0.1:6835/3958442887] [v2:127.0.0.1:6836/3958442887,v1:127.0.0.1:6837/3958442887] exists,up 51643d85-1c66-4693-bb7c-6a84ba6412e3 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:50:57.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:50:57.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:50:57.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:50:57.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:50:57.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:50:57.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:50:57.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=833673fb-e7b2-439f-8428-1c97259049f1 2024-04-25T08:50:57.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 833673fb-e7b2-439f-8428-1c97259049f1' 2024-04-25T08:50:57.183 INFO:tasks.workunit.client.0.smithi031.stdout:add osd5 833673fb-e7b2-439f-8428-1c97259049f1 2024-04-25T08:50:57.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:50:57.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxGSpm1mruDBAAE2UEJt1SlCjNc2bqf4gDrg== 2024-04-25T08:50:57.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxGSpm1mruDBAAE2UEJt1SlCjNc2bqf4gDrg=="}' 2024-04-25T08:50:57.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 833673fb-e7b2-439f-8428-1c97259049f1 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:50:57.706 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:50:57.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:50:57.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxGSpm1mruDBAAE2UEJt1SlCjNc2bqf4gDrg== --osd-uuid 833673fb-e7b2-439f-8428-1c97259049f1 2024-04-25T08:50:57.754 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.761+0000 7fb463ee77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:57.763 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.769+0000 7fb463ee77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:57.764 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.769+0000 7fb463ee77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:57.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.769+0000 7fb463ee77c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:50:57.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.773+0000 7fb463ee77c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:50:57.765 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.773+0000 7fb463ee77c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:50:57.811 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:50:57.817+0000 7fb463ee77c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:51:03.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:51:03.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:51:03.882 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd5 key to auth repository 2024-04-25T08:51:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:51:03.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:51:04.457 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.5 2024-04-25T08:51:04.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:51:04.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:51:04.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:51:04.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:51:04.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:51:04.485 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:51:04.492+0000 7fc4f71fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:04.488 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:51:04.492+0000 7fc4f71fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:04.489 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:51:04.496+0000 7fc4f71fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:04.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:51:04.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:51:04.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:51:04.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:51:04.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:51:04.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:04.936 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:51:04.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:51:04.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:04.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:05.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:51:05.060+0000 7fc4f71fd7c0 -1 Falling back to public interface 2024-04-25T08:51:05.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:06.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:06.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:06.400 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:51:06.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:51:06.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:06.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:06.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:07.553 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:51:07.556+0000 7fc4f71fd7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:51:07.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:51:07.897 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:51:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:07.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:08.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:09.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:09.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:09.340 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:51:09.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:51:09.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:09.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:09.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:10.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:10.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:10.969 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:51:10.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:51:10.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:10.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:11.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:12.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:12.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:12.474 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:51:12.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:51:12.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:12.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:12.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:13.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:13.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:13.990 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:51:13.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:51:13.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:13.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:14.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:15.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:15.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:15.571 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:51:15.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:51:15.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:15.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:16.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:17.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:17.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:17.045 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:51:17.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:51:17.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:17.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:17.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:18.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:18.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:18.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:51:18.682 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:51:18.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:18.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:19.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:20.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:20.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:20.421 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:51:20.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:51:20.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:20.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:20.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:21.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:21.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:21.887 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:51:21.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:51:21.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:21.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:22.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:23.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:23.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:23.321 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:51:23.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:51:23.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:23.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:24.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:25.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:25.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:25.286 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:51:25.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:51:25.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:25.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:26.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:27.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:27.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:27.144 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:51:27.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:51:27.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:27.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:28.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:29.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:29.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:29.077 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:51:29.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:51:29.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:29.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:30.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:31.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:31.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:31.263 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:51:31.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:51:31.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:31.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:31.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:32.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:32.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:32.858 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:51:32.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:51:32.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:32.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:33.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:34.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:34.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:34.745 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T08:51:34.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:51:34.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:34.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:35.224 INFO:tasks.workunit.client.0.smithi031.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2513254774,v1:127.0.0.1:6843/2513254774] [v2:127.0.0.1:6844/2513254774,v1:127.0.0.1:6845/2513254774] exists,up 833673fb-e7b2-439f-8428-1c97259049f1 2024-04-25T08:51:35.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:51:35.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:51:35.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:51:35.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-04-25T08:51:35.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:51:35.809 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:51:35.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:51:36.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-04-25T08:51:37.569 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T08:51:37.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-04-25T08:51:37.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:51:37.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:51:37.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:51:37.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:51:37.598 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:51:37.598 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:51:37.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:51:37.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:51:37.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:51:37.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:51:37.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:51:37.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:51:37.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:51:37.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:51:37.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:51:38.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:38.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:51:38.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836520 2024-04-25T08:51:38.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836520 2024-04-25T08:51:38.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520' 2024-04-25T08:51:38.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:38.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:51:38.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672999 2024-04-25T08:51:38.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672999 2024-04-25T08:51:38.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672999' 2024-04-25T08:51:38.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:38.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:51:38.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509468 2024-04-25T08:51:38.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509468 2024-04-25T08:51:38.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509468' 2024-04-25T08:51:38.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:38.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:51:39.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345944 2024-04-25T08:51:39.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345944 2024-04-25T08:51:39.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509468 3-85899345944' 2024-04-25T08:51:39.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:39.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:51:39.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182416 2024-04-25T08:51:39.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182416 2024-04-25T08:51:39.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509468 3-85899345944 4-107374182416' 2024-04-25T08:51:39.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:39.163 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:51:39.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018886 2024-04-25T08:51:39.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018886 2024-04-25T08:51:39.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509468 3-85899345944 4-107374182416 5-128849018886' 2024-04-25T08:51:39.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:39.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836520 2024-04-25T08:51:39.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:39.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:51:39.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836520 2024-04-25T08:51:39.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:39.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836520 2024-04-25T08:51:39.320 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836520 2024-04-25T08:51:39.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-04-25T08:51:39.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:51:39.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836518 -lt 21474836520 2024-04-25T08:51:39.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:51:40.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:51:40.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:51:41.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836520 -lt 21474836520 2024-04-25T08:51:41.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:41.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:41.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672999 2024-04-25T08:51:41.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:51:41.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672999 2024-04-25T08:51:41.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:41.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672999 2024-04-25T08:51:41.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-04-25T08:51:41.242 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672999 2024-04-25T08:51:41.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:51:41.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673000 -lt 42949672999 2024-04-25T08:51:41.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:41.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509468 2024-04-25T08:51:41.660 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:41.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:51:41.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509468 2024-04-25T08:51:41.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:41.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509468 2024-04-25T08:51:41.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-04-25T08:51:41.664 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509468 2024-04-25T08:51:41.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:51:42.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509468 2024-04-25T08:51:42.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:42.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345944 2024-04-25T08:51:42.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:42.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:51:42.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345944 2024-04-25T08:51:42.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:42.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345944 2024-04-25T08:51:42.126 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345944 2024-04-25T08:51:42.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345944' 2024-04-25T08:51:42.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:51:42.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345945 -lt 85899345944 2024-04-25T08:51:42.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:42.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:42.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182416 2024-04-25T08:51:42.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:51:42.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182416 2024-04-25T08:51:42.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:42.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182416 2024-04-25T08:51:42.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2024-04-25T08:51:42.567 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182416 2024-04-25T08:51:42.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:51:43.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182417 -lt 107374182416 2024-04-25T08:51:43.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:43.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018886 2024-04-25T08:51:43.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:43.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:51:43.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:43.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018886 2024-04-25T08:51:43.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018886 2024-04-25T08:51:43.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-04-25T08:51:43.014 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018886 2024-04-25T08:51:43.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:51:43.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018886 -lt 128849018886 2024-04-25T08:51:43.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:51:43.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:43.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:44.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:51:44.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:51:44.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:51:44.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:51:44.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:51:44.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:51:44.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:51:44.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:44.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:51:44.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:44.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:44.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:45.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:51:45.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:51:45.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:51:45.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-04-25T08:51:45.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-04-25T08:51:45.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-04-25T08:51:45.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-04-25T08:51:45.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-04-25T08:51:45.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-04-25T08:51:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-04-25T08:51:45.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:45.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-04-25T08:51:46.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-04-25T08:51:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-04-25T08:51:46.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-04-25T08:51:46.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-04-25T08:51:46.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-04-25T08:51:46.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-04-25T08:51:46.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-04-25T08:51:46.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-04-25T08:51:46.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:46.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-04-25T08:51:47.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-04-25T08:51:47.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-04-25T08:51:47.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-04-25T08:51:47.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-04-25T08:51:47.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-04-25T08:51:47.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-04-25T08:51:47.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-04-25T08:51:47.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-04-25T08:51:47.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:47.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-04-25T08:51:48.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-04-25T08:51:48.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-04-25T08:51:48.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-04-25T08:51:48.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-04-25T08:51:48.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-04-25T08:51:48.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-04-25T08:51:48.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-04-25T08:51:48.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:48.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-04-25T08:51:49.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-04-25T08:51:49.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-04-25T08:51:49.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-04-25T08:51:49.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-04-25T08:51:49.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-04-25T08:51:49.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-04-25T08:51:49.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-04-25T08:51:49.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-04-25T08:51:49.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:49.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-04-25T08:51:50.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-04-25T08:51:50.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-04-25T08:51:50.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-04-25T08:51:50.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-04-25T08:51:50.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-04-25T08:51:50.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-04-25T08:51:50.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-04-25T08:51:50.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-04-25T08:51:50.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:50.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-04-25T08:51:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-04-25T08:51:51.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-04-25T08:51:51.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-04-25T08:51:51.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-04-25T08:51:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-04-25T08:51:51.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-04-25T08:51:51.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-04-25T08:51:51.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-04-25T08:51:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:51.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-04-25T08:51:52.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-04-25T08:51:52.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-04-25T08:51:52.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-04-25T08:51:52.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-04-25T08:51:52.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-04-25T08:51:52.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-04-25T08:51:52.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-04-25T08:51:52.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-04-25T08:51:52.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:52.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-04-25T08:51:53.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-04-25T08:51:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-04-25T08:51:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-04-25T08:51:53.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-04-25T08:51:53.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-04-25T08:51:53.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-04-25T08:51:53.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-04-25T08:51:53.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:53.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-04-25T08:51:54.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-04-25T08:51:54.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-04-25T08:51:54.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-04-25T08:51:54.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-04-25T08:51:54.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-04-25T08:51:54.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-04-25T08:51:54.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-04-25T08:51:54.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:54.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-04-25T08:51:55.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-04-25T08:51:55.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-04-25T08:51:55.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-04-25T08:51:55.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-04-25T08:51:55.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-04-25T08:51:55.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-04-25T08:51:55.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-04-25T08:51:55.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-04-25T08:51:55.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:55.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-04-25T08:51:56.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-04-25T08:51:56.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-04-25T08:51:56.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-04-25T08:51:56.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-04-25T08:51:56.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-04-25T08:51:56.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-04-25T08:51:56.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-04-25T08:51:56.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-04-25T08:51:56.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:56.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-04-25T08:51:57.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-04-25T08:51:57.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-04-25T08:51:57.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-04-25T08:51:57.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-04-25T08:51:57.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-04-25T08:51:57.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-04-25T08:51:57.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-04-25T08:51:57.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-04-25T08:51:57.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:57.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-04-25T08:51:58.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-04-25T08:51:58.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-04-25T08:51:58.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-04-25T08:51:58.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-04-25T08:51:58.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-04-25T08:51:58.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-04-25T08:51:58.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-04-25T08:51:58.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:58.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-04-25T08:51:59.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-04-25T08:51:59.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-04-25T08:51:59.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-04-25T08:51:59.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-04-25T08:51:59.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-04-25T08:51:59.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-04-25T08:51:59.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-04-25T08:51:59.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-04-25T08:51:59.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:51:59.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-04-25T08:52:00.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-04-25T08:52:00.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-04-25T08:52:00.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-04-25T08:52:00.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-04-25T08:52:00.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-04-25T08:52:00.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-04-25T08:52:00.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-04-25T08:52:00.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-04-25T08:52:00.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:00.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-04-25T08:52:01.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-04-25T08:52:01.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-04-25T08:52:01.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-04-25T08:52:01.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-04-25T08:52:01.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-04-25T08:52:01.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-04-25T08:52:01.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-04-25T08:52:01.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-04-25T08:52:01.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:01.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-04-25T08:52:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-04-25T08:52:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-04-25T08:52:02.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-04-25T08:52:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-04-25T08:52:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-04-25T08:52:02.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-04-25T08:52:02.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-04-25T08:52:02.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-04-25T08:52:02.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:02.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-04-25T08:52:03.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-04-25T08:52:03.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-04-25T08:52:03.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-04-25T08:52:03.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-04-25T08:52:03.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-04-25T08:52:03.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-04-25T08:52:03.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-04-25T08:52:03.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-04-25T08:52:03.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:03.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-04-25T08:52:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-04-25T08:52:04.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-04-25T08:52:04.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-04-25T08:52:04.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-04-25T08:52:04.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-04-25T08:52:04.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-04-25T08:52:04.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-04-25T08:52:04.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:04.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-04-25T08:52:05.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-04-25T08:52:05.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-04-25T08:52:05.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-04-25T08:52:05.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-04-25T08:52:05.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-04-25T08:52:05.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-04-25T08:52:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-04-25T08:52:05.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-04-25T08:52:05.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:05.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-04-25T08:52:06.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-04-25T08:52:06.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-04-25T08:52:06.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-04-25T08:52:06.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-04-25T08:52:06.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-04-25T08:52:06.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-04-25T08:52:06.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-04-25T08:52:06.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-04-25T08:52:06.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:06.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-04-25T08:52:07.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-04-25T08:52:07.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-04-25T08:52:07.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-04-25T08:52:07.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-04-25T08:52:07.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-04-25T08:52:07.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-04-25T08:52:07.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-04-25T08:52:07.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-04-25T08:52:07.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:52:07.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-04-25T08:52:08.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T08:52:08.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:52:08.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:52:08.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:52:08.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:52:08.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-04-25T08:52:08.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-04-25T08:52:08.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:52:08.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:52:08.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:52:08.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:52:09.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-04-25T08:52:09.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-04-25T08:52:09.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:52:09.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:52:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:52:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:52:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:52:09.029 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:52:09.030 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:52:09.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:52:09.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:52:09.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:52:09.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-04-25T08:52:09.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-04-25T08:52:10.778 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:52:10.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-04-25T08:52:11.425 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already out. osd.0 is already out. 2024-04-25T08:52:11.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-04-25T08:52:12.689 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 4 2024-04-25T08:52:12.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-04-25T08:52:13.734 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:52:13.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T08:52:13.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:52:13.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:52:13.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:52:13.756 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:52:14.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-04-25T08:52:14.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T08:52:14.736 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:52:14.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-04-25T08:52:16.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-04-25T08:52:16.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:52:16.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:52:16.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:52:16.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:52:16.761 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:52:16.761 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:52:16.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:52:16.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:52:16.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:52:16.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:52:16.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:52:16.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:52:16.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:52:16.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:52:16.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:17.328 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:52:17.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836530 2024-04-25T08:52:17.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836530 2024-04-25T08:52:17.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530' 2024-04-25T08:52:17.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:17.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:52:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673009 2024-04-25T08:52:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673009 2024-04-25T08:52:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673009' 2024-04-25T08:52:17.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:17.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:52:17.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509478 2024-04-25T08:52:17.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509478 2024-04-25T08:52:17.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673009 2-64424509478' 2024-04-25T08:52:17.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:17.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:52:18.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T08:52:18.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T08:52:18.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673009 2-64424509478 3-85899345954' 2024-04-25T08:52:18.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:18.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:52:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182426 2024-04-25T08:52:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182426 2024-04-25T08:52:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673009 2-64424509478 3-85899345954 4-107374182426' 2024-04-25T08:52:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:18.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:52:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018895 2024-04-25T08:52:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018895 2024-04-25T08:52:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673009 2-64424509478 3-85899345954 4-107374182426 5-128849018895' 2024-04-25T08:52:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:18.356 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836530 2024-04-25T08:52:18.357 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:18.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:52:18.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836530 2024-04-25T08:52:18.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:18.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836530 2024-04-25T08:52:18.359 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836530 2024-04-25T08:52:18.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-25T08:52:18.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:52:18.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836530 -lt 21474836530 2024-04-25T08:52:18.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:18.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673009 2024-04-25T08:52:18.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:18.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:52:18.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673009 2024-04-25T08:52:18.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:18.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673009 2024-04-25T08:52:18.821 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673009 2024-04-25T08:52:18.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-04-25T08:52:18.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:52:19.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673009 -lt 42949673009 2024-04-25T08:52:19.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:19.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509478 2024-04-25T08:52:19.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:19.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:52:19.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509478 2024-04-25T08:52:19.331 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:19.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509478 2024-04-25T08:52:19.332 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509478 2024-04-25T08:52:19.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-04-25T08:52:19.333 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:52:19.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509478 -lt 64424509478 2024-04-25T08:52:19.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:19.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:19.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345954 2024-04-25T08:52:19.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:52:19.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345954 2024-04-25T08:52:19.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:19.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T08:52:19.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345954' 2024-04-25T08:52:19.862 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345954 2024-04-25T08:52:19.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:52:20.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345954 -lt 85899345954 2024-04-25T08:52:20.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:20.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:20.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182426 2024-04-25T08:52:20.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:52:20.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182426 2024-04-25T08:52:20.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:20.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182426 2024-04-25T08:52:20.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2024-04-25T08:52:20.340 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182426 2024-04-25T08:52:20.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:52:20.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182426 -lt 107374182426 2024-04-25T08:52:20.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:20.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018895 2024-04-25T08:52:20.927 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:20.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:52:20.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018895 2024-04-25T08:52:20.928 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:20.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018895 2024-04-25T08:52:20.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018895' 2024-04-25T08:52:20.930 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018895 2024-04-25T08:52:20.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:52:21.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018896 -lt 128849018895 2024-04-25T08:52:21.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:52:21.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:21.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:22.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:52:22.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:22.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:22.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:22.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:22.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:22.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:22.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:22.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:22.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:22.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:22.538 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:52:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:52:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:52:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:23.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:23.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:23.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:23.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:23.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:23.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:23.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:23.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:23.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:23.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:23.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:23.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:24.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:24.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:24.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-04-25T08:52:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-04-25T08:52:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:25.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:25.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:25.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:25.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:25.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:25.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:25.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:25.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:25.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:25.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:25.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:26.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:26.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:26.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:52:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:52:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:27.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:27.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:27.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:27.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:27.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:27.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:27.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:27.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:27.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:27.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:27.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:27.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:28.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:28.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:28.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:28.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:52:28.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:52:28.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:28.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:28.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:29.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:29.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:29.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:29.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:29.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:29.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:29.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:29.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:29.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:29.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:29.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:29.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:30.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:30.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:31.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:52:31.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:52:31.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:31.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:31.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:31.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:31.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:31.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:31.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:31.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:31.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:31.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:31.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:31.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:31.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:31.602 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:31.602 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:32.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:32.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:32.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:32.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:32.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:32.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:32.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:32.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:32.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:52:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:52:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:32.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:33.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:33.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:33.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:33.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:33.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:33.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:33.017 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:33.017 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:33.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:33.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:33.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:33.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:34.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:34.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:52:34.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:52:34.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:34.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:34.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:35.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:35.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:35.018 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:35.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:35.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:35.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:35.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:35.019 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:35.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:35.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:35.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:35.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:36.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:36.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:36.237 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:37.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:52:37.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:52:37.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:37.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:37.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:37.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:37.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:37.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:37.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:37.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:37.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:37.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:37.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:37.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:37.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:37.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:37.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:38.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:38.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:38.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:38.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:38.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:39.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:52:39.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:52:39.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:39.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:39.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:39.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:39.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:39.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:39.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:39.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:39.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:39.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:39.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:39.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:39.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:39.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:39.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:40.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:40.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:52:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:52:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:40.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:40.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:40.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:40.874 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:40.874 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:40.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:40.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:40.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:40.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:41.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:41.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:41.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:41.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:41.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:41.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:41.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:42.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:52:42.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:52:42.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:42.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:42.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:42.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:42.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:42.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:42.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:42.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:42.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:42.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:42.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:43.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:43.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:43.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:43.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:43.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:43.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:43.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:52:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:52:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:44.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:44.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:44.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:44.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:44.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:44.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:44.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:44.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:45.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:45.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:45.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:45.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:45.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:45.670 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:52:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:52:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:46.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:46.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:46.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:46.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:46.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:46.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:46.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:46.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:46.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:46.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:46.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:46.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:46.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:47.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:47.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:47.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:48.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:52:48.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:52:48.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:48.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:48.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:48.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:48.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:48.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:48.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:48.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:48.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:48.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:48.281 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:48.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:48.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:48.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:48.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:49.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:49.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:49.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:49.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:49.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:49.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:49.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:49.470 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:49.470 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:50.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:52:50.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:52:50.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:50.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:50.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:50.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:50.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:50.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:50.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:50.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:50.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:50.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:50.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:50.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:50.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:50.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:50.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:51.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:51.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:51.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:51.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:51.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:52:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:52:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:51.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:51.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:51.882 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:51.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:51.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:51.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:51.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:51.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:52.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:52.301 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:52.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:52.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:52.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:52.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:52.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:53.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:52:53.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:52:53.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:53.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:53.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:53.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:53.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:53.547 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:53.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:53.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:53.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:53.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:53.548 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:53.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:53.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:53.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:53.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:54.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:54.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:55.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-04-25T08:52:55.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-04-25T08:52:55.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:55.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:55.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:55.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:55.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:55.166 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:55.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:55.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:55.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:55.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:55.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:55.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:55.575 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:55.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:55.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:56.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:56.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:56.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:56.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:56.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:56.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:56.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:56.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:56.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:56.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-04-25T08:52:56.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-04-25T08:52:56.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:56.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:56.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:56.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:56.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:56.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:56.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:56.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:56.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:56.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:56.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:57.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:52:57.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:57.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:57.230 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:57.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:52:57.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:52:57.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:52:57.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T08:52:57.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-04-25T08:52:57.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-04-25T08:52:57.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:52:57.793 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:52:57.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-04-25T08:52:57.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:52:57.819 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:52:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:52:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:52:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:52:57.820 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:52:57.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-04-25T08:52:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-04-25T08:52:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:52:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:52:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:52:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:52:57.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:52:57.824 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.824 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:52:57.824 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:52:57.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:52:57.838 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:52:57.839 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:52:57.839 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:52:57.842 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:52:57.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:52:57.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:52:57.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:52:57.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:52:57.858 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:52:57.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:52:58.445 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:52:58.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-04-25T08:52:58.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:52:58.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:52:58.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:52:58.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:52:58.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:52:59.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:52:59.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:52:59.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:52:59.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:52:59.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:52:59.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:52:59.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:52:59.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:52:59.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:52:59.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:52:59.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:52:59.791 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:52:59.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:52:59.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:52:59.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:52:59.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:52:59.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:52:59.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:52:59.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:52:59.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:52:59.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:52:59.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:52:59.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:52:59.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:52:59.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:52:59.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:52:59.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:52:59.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:52:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:52:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:52:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:52:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:52:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:52:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:52:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:52:59.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:52:59.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:52:59.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:52:59.844 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:52:59.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:52:59.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:52:59.846 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:52:59.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:52:59.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:52:59.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:52:59.847 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:52:59.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:52:59.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:52:59.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:52:59.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:52:59.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:52:59.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:52:59.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:52:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:52:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:52:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:52:59.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:52:59.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:52:59.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:52:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274587 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-04-25T08:52:59.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:52:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:52:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:52:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:52:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:52:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:53:00.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:53:00.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:53:00.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:53:00.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:53:00.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:53:00.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:53:00.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:53:00.555 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:53:00.556 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:53:00.556 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:53:00.556 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:53:00.557 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:53:00.557 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:00.557 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:53:00.557 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:53:00.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:53:00.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get fsid 2024-04-25T08:53:00.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:53:00.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:53:00.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:53:00.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:53:00.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:53:00.678 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:53:00.678 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:53:00.678 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:53:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:53:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:53:00.679 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274587/ceph-mon.a.asok 2024-04-25T08:53:00.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:53:00.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274587/ceph-mon.a.asok config get mon_host 2024-04-25T08:53:00.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-04-25T08:53:00.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:53:00.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:53:00.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:53:00.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:53:00.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:53:00.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:53:01.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:53:01.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:01.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:01.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:01.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:01.304 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:53:01.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:53:01.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:53:01.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:53:01.342 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:53:01.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:01.348 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:01.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:01.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:53:01.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:01.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8af7de4-e6fe-4b32-b56a-847f36c9f1b1 2024-04-25T08:53:01.350 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 f8af7de4-e6fe-4b32-b56a-847f36c9f1b1 2024-04-25T08:53:01.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f8af7de4-e6fe-4b32-b56a-847f36c9f1b1' 2024-04-25T08:53:01.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:01.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtGSpmeY98FhAA9SUCvBJnsiouvYMCUHyeNQ== 2024-04-25T08:53:01.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtGSpmeY98FhAA9SUCvBJnsiouvYMCUHyeNQ=="}' 2024-04-25T08:53:01.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8af7de4-e6fe-4b32-b56a-847f36c9f1b1 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:53:01.675 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:53:01.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:53:01.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtGSpmeY98FhAA9SUCvBJnsiouvYMCUHyeNQ== --osd-uuid f8af7de4-e6fe-4b32-b56a-847f36c9f1b1 2024-04-25T08:53:01.732 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.739+0000 7f33155037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:01.743 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.751+0000 7f33155037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:01.744 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.755+0000 7f33155037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:01.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.755+0000 7f33155037c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:53:01.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.755+0000 7f33155037c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:53:01.745 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.755+0000 7f33155037c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:53:01.783 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:01.791+0000 7f33155037c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:53:07.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:53:07.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:53:07.896 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:53:07.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:53:07.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:53:08.481 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:53:08.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:53:08.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:53:08.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:53:08.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:53:08.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:53:08.507 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:08.514+0000 7f98f228b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:08.515 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:08.522+0000 7f98f228b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:08.516 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:08.522+0000 7f98f228b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:08.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:53:08.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:53:08.921 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:53:08.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:53:08.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:53:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:53:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:53:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:09.074 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:09.082+0000 7f98f228b7c0 -1 Falling back to public interface 2024-04-25T08:53:09.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:10.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:10.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:10.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:53:10.371 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:53:10.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:10.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:10.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:11.619 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:11.626+0000 7f98f228b7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:53:11.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:11.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:11.813 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:53:11.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:53:11.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:11.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:12.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:13.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:13.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:13.409 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:53:13.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:53:13.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:13.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:13.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:14.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:14.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:14.848 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:53:14.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:53:14.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:14.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:15.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:16.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:16.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:16.438 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:53:16.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:53:16.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:16.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:16.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:17.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:17.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:17.883 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:53:17.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:53:17.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:17.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:18.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:19.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:19.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:19.368 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:53:19.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:53:19.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:19.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:19.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:20.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:20.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:20.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:53:20.815 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:53:20.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:20.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:21.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:22.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:22.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:22.248 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:53:22.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:53:22.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:22.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:22.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:23.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:23.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:23.764 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:53:23.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:53:23.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:23.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:24.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:25.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:25.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:25.366 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:53:25.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:53:25.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:25.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:25.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:26.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:26.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:26.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:53:26.849 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:53:26.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:26.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:27.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:28.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:28.495 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:53:28.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:53:28.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:28.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:30.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:30.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:30.240 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:53:30.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:53:30.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:30.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:31.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:32.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:32.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:32.336 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:53:32.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:53:32.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:32.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:33.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:34.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:34.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:34.306 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:53:34.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:53:34.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:34.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:34.736 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3649725846,v1:127.0.0.1:6803/3649725846] [v2:127.0.0.1:6804/3649725846,v1:127.0.0.1:6805/3649725846] exists,up f8af7de4-e6fe-4b32-b56a-847f36c9f1b1 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:34.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:34.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:34.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:34.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:34.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:34.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:34.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:53:34.743 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:34.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c33dc1cc-daf3-45bb-b1e1-bcbb8d6555d4 2024-04-25T08:53:34.744 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 c33dc1cc-daf3-45bb-b1e1-bcbb8d6555d4 2024-04-25T08:53:34.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c33dc1cc-daf3-45bb-b1e1-bcbb8d6555d4' 2024-04-25T08:53:34.744 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:34.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOGipmy37jLRAAoEpELK7BcGrg7XgR7YgFbw== 2024-04-25T08:53:34.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOGipmy37jLRAAoEpELK7BcGrg7XgR7YgFbw=="}' 2024-04-25T08:53:34.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c33dc1cc-daf3-45bb-b1e1-bcbb8d6555d4 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:53:35.281 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:53:35.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:53:35.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOGipmy37jLRAAoEpELK7BcGrg7XgR7YgFbw== --osd-uuid c33dc1cc-daf3-45bb-b1e1-bcbb8d6555d4 2024-04-25T08:53:35.334 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.342+0000 7f6c736207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:35.336 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.346+0000 7f6c736207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:35.337 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.346+0000 7f6c736207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:35.338 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.346+0000 7f6c736207c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:53:35.338 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.346+0000 7f6c736207c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:53:35.338 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.346+0000 7f6c736207c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:53:35.403 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:35.410+0000 7f6c736207c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:53:41.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:53:41.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:53:41.278 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:53:41.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:53:41.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:53:41.856 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:53:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:53:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:53:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:53:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:53:41.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:53:41.889 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:41.898+0000 7efef29d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:41.916 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:41.922+0000 7efef29d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:41.917 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:41.926+0000 7efef29d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:42.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:53:42.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:53:42.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:53:42.280 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:53:42.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:53:42.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:53:42.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:42.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:53:42.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:42.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:42.474 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:42.482+0000 7efef29d47c0 -1 Falling back to public interface 2024-04-25T08:53:42.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:43.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:43.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:43.775 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:53:43.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:53:43.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:43.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:44.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:44.899 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:53:44.906+0000 7efef29d47c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:53:45.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:45.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:45.195 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:53:45.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:53:45.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:45.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:45.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:46.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:46.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:46.678 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:53:46.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:53:46.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:46.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:47.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:48.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:48.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:48.125 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:53:48.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:53:48.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:48.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:48.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:49.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:49.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:49.688 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:53:49.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:53:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:50.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:51.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:51.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:51.157 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:53:51.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:53:51.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:51.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:51.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:52.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:52.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:52.695 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:53:52.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:53:52.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:52.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:53.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:54.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:54.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:54.130 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:53:54.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:53:54.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:54.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:54.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:55.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:55.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:55.657 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:53:55.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:53:55.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:55.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:56.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:57.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:57.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:57.116 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:53:57.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:53:57.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:57.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:57.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:58.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:58.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:58.905 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:53:58.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:53:58.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:58.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:53:59.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:00.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:00.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:54:00.574 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:54:00.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:00.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:01.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:02.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:02.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:02.515 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:54:02.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:54:02.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:02.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:03.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:04.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:04.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:04.259 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:54:04.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:54:04.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:04.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:05.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:06.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:06.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:06.209 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:54:06.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:54:06.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:06.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:06.676 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1800265465,v1:127.0.0.1:6811/1800265465] [v2:127.0.0.1:6812/1800265465,v1:127.0.0.1:6813/1800265465] exists,up c33dc1cc-daf3-45bb-b1e1-bcbb8d6555d4 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:54:06.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:54:06.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:54:06.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:54:06.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:54:06.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:54:06.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:54:06.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:54:06.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0737b35e-86ed-4b9f-a8c9-43523b645b82 2024-04-25T08:54:06.684 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 0737b35e-86ed-4b9f-a8c9-43523b645b82 2024-04-25T08:54:06.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0737b35e-86ed-4b9f-a8c9-43523b645b82' 2024-04-25T08:54:06.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:54:06.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuGipmNR+ZKhAAV/iRI/0h71OhsmbFEJ6K8w== 2024-04-25T08:54:06.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuGipmNR+ZKhAAV/iRI/0h71OhsmbFEJ6K8w=="}' 2024-04-25T08:54:06.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0737b35e-86ed-4b9f-a8c9-43523b645b82 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:54:07.254 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:54:07.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:54:07.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuGipmNR+ZKhAAV/iRI/0h71OhsmbFEJ6K8w== --osd-uuid 0737b35e-86ed-4b9f-a8c9-43523b645b82 2024-04-25T08:54:07.302 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.310+0000 7f4be08317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:07.305 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.314+0000 7f4be08317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:07.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.314+0000 7f4be08317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:07.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.314+0000 7f4be08317c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:54:07.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.314+0000 7f4be08317c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:54:07.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.314+0000 7f4be08317c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:54:07.352 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:07.358+0000 7f4be08317c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:54:13.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:54:13.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:54:13.239 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:54:13.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:54:13.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:54:13.816 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:54:13.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:54:13.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6c1b7542-d9b9-475a-b9e9-62445ab14188 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274587/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:54:13.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:54:13.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:54:13.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:54:13.849 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:13.857+0000 7f67259ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:13.851 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:13.857+0000 7f67259ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:13.852 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:13.861+0000 7f67259ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:54:14.248 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:54:14.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:14.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:14.398 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:14.405+0000 7f67259ca7c0 -1 Falling back to public interface 2024-04-25T08:54:14.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:15.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:15.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:15.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:54:15.709 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:54:15.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:15.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:16.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:54:17.017+0000 7f67259ca7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:54:17.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:17.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:17.191 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:54:17.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:54:17.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:17.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:17.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:18.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:18.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:18.733 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:54:18.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:54:18.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:18.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:19.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:20.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:20.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:20.270 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:54:20.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:54:20.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:20.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:20.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:21.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:21.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:21.833 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:54:21.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:54:21.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:21.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:22.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:23.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:23.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:23.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:54:23.270 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:54:23.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:23.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:23.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:24.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:24.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:24.695 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:54:24.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:54:24.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:24.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:25.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:26.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:26.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:26.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:54:26.259 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:54:26.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:26.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:26.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:27.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:27.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:27.735 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:54:27.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:54:27.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:27.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:28.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:29.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:29.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:29.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:54:29.169 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:54:29.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:29.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:30.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:31.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:31.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:31.112 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:54:31.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:54:31.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:31.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:31.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:32.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:32.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:32.867 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:54:32.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:54:32.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:32.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:33.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:34.961 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:54:34.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:54:34.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:34.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:35.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:36.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:36.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:36.902 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:54:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:54:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:36.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:38.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:39.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:39.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:39.006 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:54:39.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:54:39.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:39.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:39.451 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4100240223,v1:127.0.0.1:6819/4100240223] [v2:127.0.0.1:6820/4100240223,v1:127.0.0.1:6821/4100240223] exists,up 0737b35e-86ed-4b9f-a8c9-43523b645b82 2024-04-25T08:54:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:54:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:54:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:54:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-04-25T08:54:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:54:40.004 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T08:54:40.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:54:41.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-04-25T08:54:41.918 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 1 2024-04-25T08:54:41.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-04-25T08:54:41.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:54:41.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:54:41.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:54:41.946 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:54:41.947 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:54:41.947 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:54:41.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:54:41.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:54:41.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:54:42.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:54:42.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:54:42.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:54:42.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:54:42.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:54:42.078 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:54:42.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:54:42.618 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:54:42.618 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T08:54:42.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:54:42.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:42.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:54:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T08:54:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T08:54:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T08:54:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:42.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:54:42.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672974 2024-04-25T08:54:42.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672974 2024-04-25T08:54:42.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-04-25T08:54:42.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:42.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:54:43.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542149 2024-04-25T08:54:43.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542149 2024-04-25T08:54:43.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-60129542149' 2024-04-25T08:54:43.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:43.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T08:54:43.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:43.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:54:43.100 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T08:54:43.100 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:43.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T08:54:43.101 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836500 2024-04-25T08:54:43.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T08:54:43.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:43.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836500 2024-04-25T08:54:43.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:54:44.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:54:44.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:44.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836500 2024-04-25T08:54:44.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:44.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672974 2024-04-25T08:54:44.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:44.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:54:44.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672974 2024-04-25T08:54:44.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:44.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672974 2024-04-25T08:54:44.987 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672974 2024-04-25T08:54:44.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-04-25T08:54:44.988 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:54:45.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672974 -lt 42949672974 2024-04-25T08:54:45.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:45.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542149 2024-04-25T08:54:45.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:45.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:54:45.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542149 2024-04-25T08:54:45.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:45.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542149 2024-04-25T08:54:45.448 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 60129542149 2024-04-25T08:54:45.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2024-04-25T08:54:45.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:54:45.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542149 -lt 60129542149 2024-04-25T08:54:45.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:54:45.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:45.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:46.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:54:46.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:46.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:46.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:46.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:46.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:46.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:46.460 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:46.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:54:46.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:46.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:46.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:47.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:54:47.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:54:47.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:54:47.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-04-25T08:54:47.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-04-25T08:54:47.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-04-25T08:54:47.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-04-25T08:54:47.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-04-25T08:54:47.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-04-25T08:54:47.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-04-25T08:54:47.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:47.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-04-25T08:54:48.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-04-25T08:54:48.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-04-25T08:54:48.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-04-25T08:54:48.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-04-25T08:54:48.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-04-25T08:54:48.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-04-25T08:54:48.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-04-25T08:54:48.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-04-25T08:54:48.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-04-25T08:54:48.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-04-25T08:54:48.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-04-25T08:54:48.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:48.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-04-25T08:54:49.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-04-25T08:54:49.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-04-25T08:54:49.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-04-25T08:54:49.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-04-25T08:54:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-04-25T08:54:49.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-04-25T08:54:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-04-25T08:54:49.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-04-25T08:54:49.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-04-25T08:54:49.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-04-25T08:54:49.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-04-25T08:54:49.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-04-25T08:54:49.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-04-25T08:54:49.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-04-25T08:54:49.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:49.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-04-25T08:54:50.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-04-25T08:54:50.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-04-25T08:54:50.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-04-25T08:54:50.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-04-25T08:54:50.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-04-25T08:54:50.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-04-25T08:54:50.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-04-25T08:54:50.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-04-25T08:54:50.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-04-25T08:54:50.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-04-25T08:54:50.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-04-25T08:54:50.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:50.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-04-25T08:54:51.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-04-25T08:54:51.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-04-25T08:54:51.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-04-25T08:54:51.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-04-25T08:54:51.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-04-25T08:54:51.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-04-25T08:54:51.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-04-25T08:54:51.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-04-25T08:54:51.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-04-25T08:54:51.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-04-25T08:54:51.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-04-25T08:54:51.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-04-25T08:54:51.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:51.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-04-25T08:54:52.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-04-25T08:54:52.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-04-25T08:54:52.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-04-25T08:54:52.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-04-25T08:54:52.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-04-25T08:54:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-04-25T08:54:52.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-04-25T08:54:52.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-04-25T08:54:52.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-04-25T08:54:52.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-04-25T08:54:52.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-04-25T08:54:52.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:52.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-04-25T08:54:53.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-04-25T08:54:53.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-04-25T08:54:53.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-04-25T08:54:53.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-04-25T08:54:53.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-04-25T08:54:53.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-04-25T08:54:53.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-04-25T08:54:53.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-04-25T08:54:53.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-04-25T08:54:53.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-04-25T08:54:53.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-04-25T08:54:53.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-04-25T08:54:53.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:53.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-04-25T08:54:54.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-04-25T08:54:54.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-04-25T08:54:54.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-04-25T08:54:54.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-04-25T08:54:54.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-04-25T08:54:54.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-04-25T08:54:54.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-04-25T08:54:54.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-04-25T08:54:54.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-04-25T08:54:54.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-04-25T08:54:54.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-04-25T08:54:54.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:54.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-04-25T08:54:55.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-04-25T08:54:55.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-04-25T08:54:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-04-25T08:54:55.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-04-25T08:54:55.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-04-25T08:54:55.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-04-25T08:54:55.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-04-25T08:54:55.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-04-25T08:54:55.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-04-25T08:54:55.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-04-25T08:54:55.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-04-25T08:54:55.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-04-25T08:54:55.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-04-25T08:54:55.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:55.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-04-25T08:54:56.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-04-25T08:54:56.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-04-25T08:54:56.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-04-25T08:54:56.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-04-25T08:54:56.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-04-25T08:54:56.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-04-25T08:54:56.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-04-25T08:54:56.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-04-25T08:54:56.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-04-25T08:54:56.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-04-25T08:54:56.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-04-25T08:54:56.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-04-25T08:54:56.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:56.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-04-25T08:54:57.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-04-25T08:54:57.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-04-25T08:54:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-04-25T08:54:57.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-04-25T08:54:57.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-04-25T08:54:57.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-04-25T08:54:57.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-04-25T08:54:57.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-04-25T08:54:57.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-04-25T08:54:57.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-04-25T08:54:57.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-04-25T08:54:57.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-04-25T08:54:57.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-04-25T08:54:57.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-04-25T08:54:57.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:57.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-04-25T08:54:58.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-04-25T08:54:58.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-04-25T08:54:58.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-04-25T08:54:58.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-04-25T08:54:58.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-04-25T08:54:58.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-04-25T08:54:58.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-04-25T08:54:58.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-04-25T08:54:58.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-04-25T08:54:58.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-04-25T08:54:58.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-04-25T08:54:58.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-04-25T08:54:58.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:58.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-04-25T08:54:59.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-04-25T08:54:59.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-04-25T08:54:59.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-04-25T08:54:59.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-04-25T08:54:59.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-04-25T08:54:59.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-04-25T08:54:59.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-04-25T08:54:59.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-04-25T08:54:59.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-04-25T08:54:59.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-04-25T08:54:59.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-04-25T08:54:59.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-04-25T08:54:59.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-04-25T08:54:59.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:54:59.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-04-25T08:55:00.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-04-25T08:55:00.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-04-25T08:55:00.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-04-25T08:55:00.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-04-25T08:55:00.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-04-25T08:55:00.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-04-25T08:55:00.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-04-25T08:55:00.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-04-25T08:55:00.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-04-25T08:55:00.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-04-25T08:55:00.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-04-25T08:55:00.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-04-25T08:55:00.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-04-25T08:55:00.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:00.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-04-25T08:55:01.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-04-25T08:55:01.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-04-25T08:55:01.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-04-25T08:55:01.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-04-25T08:55:01.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-04-25T08:55:01.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-04-25T08:55:01.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-04-25T08:55:01.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-04-25T08:55:01.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.580 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-04-25T08:55:01.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-04-25T08:55:01.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-04-25T08:55:01.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-04-25T08:55:01.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-04-25T08:55:01.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-04-25T08:55:01.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:01.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-04-25T08:55:02.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:02.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-04-25T08:55:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-04-25T08:55:02.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:02.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-04-25T08:55:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-04-25T08:55:02.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:55:02.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-04-25T08:55:02.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-04-25T08:55:02.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:55:02.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:55:02.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:55:02.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:55:02.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-04-25T08:55:02.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-04-25T08:55:02.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:55:02.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:55:02.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:55:02.942 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:55:03.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-04-25T08:55:03.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-04-25T08:55:04.259 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is set 2024-04-25T08:55:04.285 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-04-25T08:55:04.287 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-04-25T08:55:04.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:55:04.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-04-25T08:55:04.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-04-25T08:55:04.883 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T08:55:04.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-04-25T08:55:04.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-04-25T08:55:05.623 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 4 2024-04-25T08:55:05.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-04-25T08:55:06.544 INFO:tasks.workunit.client.0.smithi031.stderr:norecover is unset 2024-04-25T08:55:06.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-04-25T08:55:06.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:55:06.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:55:06.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:55:06.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:55:07.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:55:07.360 INFO:tasks.workunit.client.0.smithi031.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:55:07.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-04-25T08:55:17.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-04-25T08:55:17.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:55:17.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:55:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:55:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:55:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T08:55:17.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:55:17.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:17.840 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:55:18.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-04-25T08:55:18.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-04-25T08:55:18.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-04-25T08:55:18.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:18.034 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:55:18.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T08:55:18.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T08:55:18.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672983' 2024-04-25T08:55:18.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:18.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:55:18.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542158 2024-04-25T08:55:18.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542158 2024-04-25T08:55:18.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672983 2-60129542158' 2024-04-25T08:55:18.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:18.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-04-25T08:55:18.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:18.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:55:18.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-04-25T08:55:18.342 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-04-25T08:55:18.343 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836509 2024-04-25T08:55:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-25T08:55:18.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:55:18.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-04-25T08:55:18.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:18.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:18.792 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T08:55:18.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:55:18.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T08:55:18.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:18.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T08:55:18.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T08:55:18.796 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672983 2024-04-25T08:55:18.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:19.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672983 2024-04-25T08:55:19.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:55:20.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:55:20.267 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:20.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T08:55:20.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:20.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542158 2024-04-25T08:55:20.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:20.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:55:20.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542158 2024-04-25T08:55:20.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:20.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542158 2024-04-25T08:55:20.741 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 60129542158 2024-04-25T08:55:20.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2024-04-25T08:55:20.742 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:55:21.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542159 -lt 60129542158 2024-04-25T08:55:21.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-04-25T08:55:21.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-04-25T08:55:21.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-04-25T08:55:21.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-04-25T08:55:21.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-04-25T08:55:21.596 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T08:55:21.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-04-25T08:55:21.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-04-25T08:55:21.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:55:21.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:55:21.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:55:21.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:55:22.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-04-25T08:55:22.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-04-25T08:55:22.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-04-25T08:55:22.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:55:22.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:55:22.037 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:55:22.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-04-25T08:55:22.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:55:22.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:22.047 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.047 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-04-25T08:55:22.047 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:55:22.048 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:55:22.048 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:22.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-04-25T08:55:22.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-04-25T08:55:22.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-04-25T08:55:22.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:55:22.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-04-25T08:55:22.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:55:22.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:22.054 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.054 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:55:22.054 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:55:22.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:55:22.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:22.064 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.064 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-04-25T08:55:22.065 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:55:22.065 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:55:22.065 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:22.069 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:55:22.070 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:55:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:55:22.789 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T08:55:22.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-04-25T08:55:22.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:55:22.808 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:55:22.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:55:22.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:55:22.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:55:28.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:55:28.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:55:28.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:55:28.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:55:28.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:55:28.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:55:28.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:55:28.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:55:28.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:55:28.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:55:28.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:55:28.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:28.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:55:28.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:55:28.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:28.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:55:28.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:55:28.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:55:28.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:55:28.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:28.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:55:28.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:55:28.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:55:28.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:55:28.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:55:28.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-stats 0 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:55:28.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:55:28.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:55:28.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:55:28.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:55:28.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:55:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:55:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:55:28.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:55:28.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:28.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:55:28.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:55:28.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:28.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:55:28.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:55:28.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:55:28.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:55:28.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:28.179 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274587 2024-04-25T08:55:28.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274587 2024-04-25T08:55:28.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:55:28.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:55:28.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:55:28.181 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:55:28.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:55:28.193 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-04-25T08:55:28.193 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-04-25T08:55:28.246 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:55:28.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-04-25T08:55:28.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-04-25T08:55:28.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none ' 2024-04-25T08:55:28.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-04-25T08:55:28.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-04-25T08:55:28.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_rados_repair_warning 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_rep_backfill_unfound 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_rep_recovery_unfound' 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:55:28.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:55:28.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:55:28.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:55:28.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:55:28.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:55:28.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:55:28.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:55:28.261 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:55:28.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:28.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:55:28.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:28.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:55:28.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:55:28.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:55:28.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:55:28.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:28.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:28.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T08:55:28.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:55:28.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:55:28.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:55:28.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:28.268 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:28.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383712 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T08:55:28.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T08:55:28.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:55:28.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:28.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:28.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:28.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:28.740 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:28.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:55:28.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T08:55:28.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:55:28.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:55:28.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:55:28.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:55:28.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:55:28.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:55:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:55:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:55:28.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:55:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:55:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T08:55:28.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:55:28.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get fsid 2024-04-25T08:55:29.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:55:29.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:55:29.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:55:29.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:55:29.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:55:29.093 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:55:29.093 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:55:29.093 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:55:29.093 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:55:29.093 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:29.094 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:29.094 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T08:55:29.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:55:29.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get mon_host 2024-04-25T08:55:29.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T08:55:29.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:29.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:55:29.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:55:29.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:55:29.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T08:55:29.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:55:30.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:55:30.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:30.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:30.137 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:30.137 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:30.137 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:30.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:55:30.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:55:30.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:55:30.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T08:55:30.520 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T08:55:30.535 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T08:55:30.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T08:55:30.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:55:30.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T08:55:30.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:55:30.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:55:30.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:55:30.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T08:55:30.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:55:30.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0340ee48-d1f5-4995-a505-2e681da5c4ff 2024-04-25T08:55:30.544 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 0340ee48-d1f5-4995-a505-2e681da5c4ff 2024-04-25T08:55:30.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0340ee48-d1f5-4995-a505-2e681da5c4ff' 2024-04-25T08:55:30.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:55:30.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCGipmqiM9IhAAEg5jBkRMbN207h6WIGCYng== 2024-04-25T08:55:30.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCGipmqiM9IhAAEg5jBkRMbN207h6WIGCYng=="}' 2024-04-25T08:55:30.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0340ee48-d1f5-4995-a505-2e681da5c4ff -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:55:30.855 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:55:30.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:55:30.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCGipmqiM9IhAAEg5jBkRMbN207h6WIGCYng== --osd-uuid 0340ee48-d1f5-4995-a505-2e681da5c4ff 2024-04-25T08:55:30.909 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.916+0000 7fcde542e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:30.923 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.928+0000 7fcde542e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:30.924 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.932+0000 7fcde542e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:30.925 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.932+0000 7fcde542e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:55:30.925 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.932+0000 7fcde542e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:55:30.925 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.932+0000 7fcde542e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:55:30.963 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:30.968+0000 7fcde542e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T08:55:36.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T08:55:36.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:55:36.804 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T08:55:36.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:55:36.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:55:37.434 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T08:55:37.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:55:37.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:55:37.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:55:37.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:55:37.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:55:37.465 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:37.472+0000 7f1cd64bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:37.496 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:37.504+0000 7f1cd64bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:37.498 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:37.504+0000 7f1cd64bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:37.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:55:37.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:55:37.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:55:37.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:55:37.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:55:37.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:37.929 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:55:37.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:55:37.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:37.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:38.097 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:38.104+0000 7f1cd64bf7c0 -1 Falling back to public interface 2024-04-25T08:55:38.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:39.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:39.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:39.379 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:55:39.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:55:39.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:39.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:39.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:40.603 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:55:40.608+0000 7f1cd64bf7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:55:40.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:40.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:40.829 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:55:40.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:55:40.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:40.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:41.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:42.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:42.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:42.365 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:55:42.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:55:42.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:42.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:42.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:43.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:43.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:43.804 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:55:43.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:55:43.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:43.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:45.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:45.363 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:55:45.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:45.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:55:45.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:45.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:46.801 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:55:46.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:55:46.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:46.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:47.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:48.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:48.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:48.406 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:55:48.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:55:48.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:48.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:48.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:49.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:49.862 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:55:49.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:49.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:55:49.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:49.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:50.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:51.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:51.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:51.409 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:55:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:55:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:52.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:53.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:53.037 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:55:53.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:53.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:55:53.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:53.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:53.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:54.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:54.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:54.568 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:55:54.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:55:54.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:54.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:55.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:56.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:56.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:56.872 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:55:56.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:55:56.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:56.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:57.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:58.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:58.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:58.859 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:55:58.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:55:58.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:58.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:59.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:00.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:00.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:00.983 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:56:00.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:56:00.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:00.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:01.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:02.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:02.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:02.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:56:02.938 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:56:02.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:02.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:03.417 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1003314867,v1:127.0.0.1:6803/1003314867] [v2:127.0.0.1:6804/1003314867,v1:127.0.0.1:6805/1003314867] exists,up 0340ee48-d1f5-4995-a505-2e681da5c4ff 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:56:03.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:03.420 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:03.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:03.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:03.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:56:03.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T08:56:03.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:56:03.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=973ea054-a144-451c-81cf-4f6dc7bfe378 2024-04-25T08:56:03.424 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 973ea054-a144-451c-81cf-4f6dc7bfe378 2024-04-25T08:56:03.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 973ea054-a144-451c-81cf-4f6dc7bfe378' 2024-04-25T08:56:03.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:56:03.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjGipmvidIGxAAgrraw73uGEvB84cGAVRl1g== 2024-04-25T08:56:03.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjGipmvidIGxAAgrraw73uGEvB84cGAVRl1g=="}' 2024-04-25T08:56:03.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 973ea054-a144-451c-81cf-4f6dc7bfe378 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:56:03.986 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:56:04.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:56:04.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjGipmvidIGxAAgrraw73uGEvB84cGAVRl1g== --osd-uuid 973ea054-a144-451c-81cf-4f6dc7bfe378 2024-04-25T08:56:04.039 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.048+0000 7f9db05567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:04.075 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.084+0000 7f9db05567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:04.076 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.088+0000 7f9db05567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:04.077 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.088+0000 7f9db05567c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:56:04.077 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.088+0000 7f9db05567c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:56:04.077 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.088+0000 7f9db05567c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:56:04.166 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:04.176+0000 7f9db05567c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T08:56:10.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T08:56:10.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:56:10.079 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T08:56:10.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:56:10.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:56:10.651 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T08:56:10.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:56:10.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:56:10.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:56:10.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:56:10.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:56:10.676 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:10.684+0000 7f0508fdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:10.708+0000 7f0508fdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:10.698 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:10.708+0000 7f0508fdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:11.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:56:11.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:56:11.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:56:11.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:56:11.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:56:11.081 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:56:11.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:11.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:56:11.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:11.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:11.257 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:11.268+0000 7f0508fdc7c0 -1 Falling back to public interface 2024-04-25T08:56:11.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:12.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:12.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:12.524 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:56:12.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:12.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:12.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:12.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:13.853 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:13.864+0000 7f0508fdc7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:56:13.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:13.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:13.948 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:56:13.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:13.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:13.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:14.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:15.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:15.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:15.545 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:56:15.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:15.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:15.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:16.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:17.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:17.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:17.108 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:56:17.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:17.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:17.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:17.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:18.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:18.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:18.621 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:56:18.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:18.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:18.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:19.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:20.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:20.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:20.063 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:56:20.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:20.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:20.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:20.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:21.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:21.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:21.502 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:56:21.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:21.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:21.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:21.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:22.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:22.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:22.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:22.943 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:56:22.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:22.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:23.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:24.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:24.557 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:56:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:24.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:25.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:25.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:25.983 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:56:25.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:56:25.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:25.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:26.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:27.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:27.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:27.420 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:56:27.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:56:27.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:27.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:27.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:28.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:28.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:28.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:56:28.999 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:56:28.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:28.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:29.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:30.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:30.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:30.951 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:56:30.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:56:30.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:30.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:32.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:33.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:33.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:33.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:56:33.055 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:56:33.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:33.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:33.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:34.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:34.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:34.773 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:56:34.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:56:34.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:34.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:35.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:36.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:36.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:36.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:56:36.813 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:56:36.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:36.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:37.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:38.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:38.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:38.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:56:38.268 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:56:38.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:38.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:38.732 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4172694295,v1:127.0.0.1:6811/4172694295] [v2:127.0.0.1:6812/4172694295,v1:127.0.0.1:6813/4172694295] exists,up 973ea054-a144-451c-81cf-4f6dc7bfe378 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:56:38.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:56:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:56:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T08:56:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T08:56:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:56:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:56:38.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:56:38.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T08:56:38.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:56:38.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15c56d19-20e8-4cc2-af3a-68ab26975bee 2024-04-25T08:56:38.739 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 15c56d19-20e8-4cc2-af3a-68ab26975bee 2024-04-25T08:56:38.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 15c56d19-20e8-4cc2-af3a-68ab26975bee' 2024-04-25T08:56:38.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:56:38.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGGipm+WsELhAAWwfm5795hVB1Io2MtGccZA== 2024-04-25T08:56:38.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGGipm+WsELhAAWwfm5795hVB1Io2MtGccZA=="}' 2024-04-25T08:56:38.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15c56d19-20e8-4cc2-af3a-68ab26975bee -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:56:39.326 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:56:39.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:56:39.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGGipm+WsELhAAWwfm5795hVB1Io2MtGccZA== --osd-uuid 15c56d19-20e8-4cc2-af3a-68ab26975bee 2024-04-25T08:56:39.376 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.387+0000 7fa28201d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:39.451 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.459+0000 7fa28201d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:39.453 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.463+0000 7fa28201d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:39.453 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.463+0000 7fa28201d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:56:39.454 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.463+0000 7fa28201d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:56:39.454 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.463+0000 7fa28201d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:56:39.588 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:39.599+0000 7fa28201d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T08:56:45.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T08:56:45.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:56:45.135 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T08:56:45.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:56:45.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:56:45.703 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T08:56:45.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:56:45.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:56:45.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:56:45.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:56:45.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:56:45.735 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:45.743+0000 7ff94ff207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:45.761 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:45.771+0000 7ff94ff207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:45.762 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:45.771+0000 7ff94ff207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:46.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:56:46.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:56:46.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:56:46.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:56:46.135 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:56:46.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:56:46.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:46.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:56:46.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:46.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:46.329 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:46.339+0000 7ff94ff207c0 -1 Falling back to public interface 2024-04-25T08:56:46.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:47.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:47.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:47.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:47.636 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:56:47.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:47.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:48.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:48.946 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:56:48.955+0000 7ff94ff207c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:56:49.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:49.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:49.066 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:56:49.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:49.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:49.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:49.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:50.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:50.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:50.617 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:56:50.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:50.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:50.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:51.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:52.109 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:56:52.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:52.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:52.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:52.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:52.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:52.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:53.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:53.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:53.538 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:56:53.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:53.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:53.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:53.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:55.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:55.001 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:56:55.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:55.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:55.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:55.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:55.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:56.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:56.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:56.558 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:56:56.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:56.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:56.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:57.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:58.109 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:56:58.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:58.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:58.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:59.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:59.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:59.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:59.553 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:56:59.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:59.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:00.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:01.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:01.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:01.020 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:57:01.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:57:01.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:01.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:01.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:02.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:02.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:02.589 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:57:02.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:57:02.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:02.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:03.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:04.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:04.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:04.696 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:57:04.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:57:04.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:04.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:05.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:06.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:06.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:06.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:57:06.455 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:57:06.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:06.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:07.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:08.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:08.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:08.278 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:57:08.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:57:08.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:08.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:08.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:09.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:09.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:09.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:57:09.935 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:57:09.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:09.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:10.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:11.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:11.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:11.686 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:57:11.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:57:11.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:11.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:12.217 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1355598743,v1:127.0.0.1:6819/1355598743] [v2:127.0.0.1:6820/1355598743,v1:127.0.0.1:6821/1355598743] exists,up 15c56d19-20e8-4cc2-af3a-68ab26975bee 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:57:12.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:57:12.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:57:12.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:57:12.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:57:12.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:57:12.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:57:12.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T08:57:12.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:57:12.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04121504-b0eb-4008-8fc1-8c636c36c097 2024-04-25T08:57:12.225 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 04121504-b0eb-4008-8fc1-8c636c36c097 2024-04-25T08:57:12.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 04121504-b0eb-4008-8fc1-8c636c36c097' 2024-04-25T08:57:12.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:57:12.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoGipmWwhBDxAAtexkAX8bx9uRzUPIPfVXpA== 2024-04-25T08:57:12.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoGipmWwhBDxAAtexkAX8bx9uRzUPIPfVXpA=="}' 2024-04-25T08:57:12.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04121504-b0eb-4008-8fc1-8c636c36c097 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T08:57:12.749 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:57:12.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T08:57:12.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoGipmWwhBDxAAtexkAX8bx9uRzUPIPfVXpA== --osd-uuid 04121504-b0eb-4008-8fc1-8c636c36c097 2024-04-25T08:57:12.800 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.811+0000 7fe7822ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.811+0000 7fe7822ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:12.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.815+0000 7fe7822ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:12.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.815+0000 7fe7822ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:57:12.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.815+0000 7fe7822ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:57:12.804 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.815+0000 7fe7822ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:57:12.934 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:12.943+0000 7fe7822ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T08:57:19.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T08:57:19.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:57:19.463 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T08:57:19.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:57:19.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:57:20.027 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T08:57:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:57:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:57:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:57:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:57:20.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:57:20.057 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:20.066+0000 7f7d4f2c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:20.060 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:20.070+0000 7f7d4f2c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:20.061 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:20.070+0000 7f7d4f2c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:57:20.456 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T08:57:20.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:20.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:20.629 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:20.638+0000 7f7d4f2c57c0 -1 Falling back to public interface 2024-04-25T08:57:20.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:21.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:21.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:21.959 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T08:57:21.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:57:21.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:21.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:22.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:23.253 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T08:57:23.262+0000 7f7d4f2c57c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:57:23.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:23.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:23.383 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T08:57:23.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:57:23.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:23.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:23.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:24.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:24.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:24.934 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T08:57:24.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:57:24.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:24.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:25.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:26.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:26.560 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T08:57:26.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:26.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:57:26.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:26.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:27.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:28.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:28.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:28.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:57:28.044 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T08:57:28.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:28.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:28.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:29.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:29.496 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T08:57:29.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:57:29.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:29.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:30.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:31.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:31.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:31.033 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T08:57:31.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:57:31.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:31.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:31.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:32.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:32.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:32.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:57:32.530 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T08:57:32.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:32.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:33.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:34.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:34.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:34.017 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T08:57:34.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:57:34.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:34.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:34.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:35.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:35.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:35.462 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T08:57:35.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:57:35.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:35.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:36.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:37.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:37.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:37.449 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T08:57:37.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:57:37.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:37.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:38.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:39.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:39.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:39.282 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T08:57:39.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:57:39.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:39.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:39.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:40.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:40.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:40.943 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T08:57:40.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:57:40.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:40.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:41.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:42.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:42.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:42.696 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T08:57:42.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:57:42.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:42.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:43.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:44.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:44.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:44.444 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T08:57:44.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:57:44.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:44.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:45.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:46.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:46.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:46.247 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T08:57:46.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:57:46.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:46.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:46.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:47.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:47.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:47.869 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T08:57:47.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:57:47.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:47.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:57:48.548 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 32 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3863315329,v1:127.0.0.1:6827/3863315329] [v2:127.0.0.1:6828/3863315329,v1:127.0.0.1:6829/3863315329] exists,up 04121504-b0eb-4008-8fc1-8c636c36c097 2024-04-25T08:57:48.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:57:48.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:57:48.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:57:48.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T08:57:48.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:57:48.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:57:48.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:57:48.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:57:48.550 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:57:48.550 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:57:48.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:57:48.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:57:48.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:57:48.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:57:48.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:57:48.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:57:48.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:57:48.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:57:48.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:57:49.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:57:49.198 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:57:49.198 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:57:49.198 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:57:49.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:57:49.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:49.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803800 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803800 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800' 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:57:49.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574867 2024-04-25T08:57:49.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574867 2024-04-25T08:57:49.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574867' 2024-04-25T08:57:49.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:49.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:57:49.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345932 2024-04-25T08:57:49.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345932 2024-04-25T08:57:49.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574867 2-85899345932' 2024-04-25T08:57:49.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:49.729 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:57:49.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953481 2024-04-25T08:57:49.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953481 2024-04-25T08:57:49.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574867 2-85899345932 3-137438953481' 2024-04-25T08:57:49.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:49.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803800 2024-04-25T08:57:49.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:49.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:57:49.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803800 2024-04-25T08:57:49.885 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:49.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803800 2024-04-25T08:57:49.886 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803800 2024-04-25T08:57:49.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803800' 2024-04-25T08:57:49.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:50.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803798 -lt 25769803800 2024-04-25T08:57:50.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:57:51.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:57:51.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:51.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803800 -lt 25769803800 2024-04-25T08:57:51.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:51.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574867 2024-04-25T08:57:51.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:51.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:57:51.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574867 2024-04-25T08:57:51.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:51.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574867 2024-04-25T08:57:51.762 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574867 2024-04-25T08:57:51.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574867' 2024-04-25T08:57:51.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:52.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574867 -lt 55834574867 2024-04-25T08:57:52.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:52.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345932 2024-04-25T08:57:52.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:52.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:57:52.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345932 2024-04-25T08:57:52.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:52.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345932 2024-04-25T08:57:52.184 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345932 2024-04-25T08:57:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345932' 2024-04-25T08:57:52.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:57:52.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345932 -lt 85899345932 2024-04-25T08:57:52.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:52.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953481 2024-04-25T08:57:52.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:52.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:57:52.604 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:52.604 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953481 2024-04-25T08:57:52.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953481 2024-04-25T08:57:52.605 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953481 2024-04-25T08:57:52.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953481' 2024-04-25T08:57:52.605 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:57:53.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953481 -lt 137438953481 2024-04-25T08:57:53.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:57:53.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:53.021 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:53.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T08:57:53.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:53.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:53.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:53.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:53.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:53.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:53.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:57:54.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:57:54.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:57:54.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:54.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:54.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T08:57:54.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:57:54.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:57:54.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-04-25T08:57:54.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-04-25T08:57:54.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T08:57:55.082 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'pool-rep' already exists 2024-04-25T08:57:55.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:57:56.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-04-25T08:57:56.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:57:56.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:57:56.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:57:56.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:57:56.106 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:57:56.106 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:57:56.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:57:56.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:57:56.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:57:56.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:57:56.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:57:56.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:57:56.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:57:56.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:57:56.228 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:56.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:57:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803803 2024-04-25T08:57:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803803 2024-04-25T08:57:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803' 2024-04-25T08:57:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:56.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:57:57.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574870 2024-04-25T08:57:57.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574870 2024-04-25T08:57:57.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803 1-55834574870' 2024-04-25T08:57:57.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:57.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:57:57.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345935 2024-04-25T08:57:57.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345935 2024-04-25T08:57:57.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803 1-55834574870 2-85899345935' 2024-04-25T08:57:57.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:57.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:57:57.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953484 2024-04-25T08:57:57.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953484 2024-04-25T08:57:57.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803 1-55834574870 2-85899345935 3-137438953484' 2024-04-25T08:57:57.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:57.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:57.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803803 2024-04-25T08:57:57.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:57:57.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803803 2024-04-25T08:57:57.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:57.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803803 2024-04-25T08:57:57.339 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803803 2024-04-25T08:57:57.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-04-25T08:57:57.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:57.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803803 -lt 25769803803 2024-04-25T08:57:57.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:57.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574870 2024-04-25T08:57:57.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:57.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:57:57.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574870 2024-04-25T08:57:57.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:57.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574870 2024-04-25T08:57:57.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574870' 2024-04-25T08:57:57.763 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574870 2024-04-25T08:57:57.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:58.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574868 -lt 55834574870 2024-04-25T08:57:58.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:57:59.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:57:59.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:59.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574871 -lt 55834574870 2024-04-25T08:57:59.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:59.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345935 2024-04-25T08:57:59.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:59.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:57:59.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345935 2024-04-25T08:57:59.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:59.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345935 2024-04-25T08:57:59.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345935' 2024-04-25T08:57:59.711 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345935 2024-04-25T08:57:59.711 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:00.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345936 -lt 85899345935 2024-04-25T08:58:00.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:00.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953484 2024-04-25T08:58:00.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:00.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:00.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953484 2024-04-25T08:58:00.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:00.142 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953484 2024-04-25T08:58:00.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953484 2024-04-25T08:58:00.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953484' 2024-04-25T08:58:00.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:00.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953485 -lt 137438953484 2024-04-25T08:58:00.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:58:00.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:00.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:01.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:58:01.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:01.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:01.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:01.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:01.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:01.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:01.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:01.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:58:01.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:01.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:01.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:02.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-383712 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-383712 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-383712 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:02.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-383712 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-383712 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:58:02.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:58:02.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:58:02.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:58:02.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-383712 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 0 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:02.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:02.697 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:02.698 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:02.698 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:03.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:03.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:58:03.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:03.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:03.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:03.138 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:03.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:03.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:03.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:03.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:03.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:58:03.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:03.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:03.264 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:03.264 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:03.264 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:03.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:03.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:03.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383712 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383712 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:58:03.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-383712 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:03.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:03.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:58:03.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:58:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:58:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:58:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:58:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:58:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:03.560 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:04.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:04.025 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:04.025 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:58:04.025 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:58:04.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:04.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:04.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:04.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803807 2024-04-25T08:58:04.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803807 2024-04-25T08:58:04.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807' 2024-04-25T08:58:04.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:04.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:04.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574874 2024-04-25T08:58:04.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574874 2024-04-25T08:58:04.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807 1-55834574874' 2024-04-25T08:58:04.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:04.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:04.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345939 2024-04-25T08:58:04.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345939 2024-04-25T08:58:04.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807 1-55834574874 2-85899345939' 2024-04-25T08:58:04.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:04.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:58:04.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953488 2024-04-25T08:58:04.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953488 2024-04-25T08:58:04.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807 1-55834574874 2-85899345939 3-137438953488' 2024-04-25T08:58:04.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:04.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803807 2024-04-25T08:58:04.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:04.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:04.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803807 2024-04-25T08:58:04.705 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:04.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803807 2024-04-25T08:58:04.706 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803807 2024-04-25T08:58:04.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-04-25T08:58:04.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:05.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803807 -lt 25769803807 2024-04-25T08:58:05.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:05.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574874 2024-04-25T08:58:05.149 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:05.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:05.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574874 2024-04-25T08:58:05.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:05.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574874 2024-04-25T08:58:05.153 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574874 2024-04-25T08:58:05.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-04-25T08:58:05.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:05.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574874 -lt 55834574874 2024-04-25T08:58:05.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:05.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345939 2024-04-25T08:58:05.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:05.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:05.605 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345939 2024-04-25T08:58:05.605 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:05.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345939 2024-04-25T08:58:05.607 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345939 2024-04-25T08:58:05.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-04-25T08:58:05.607 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:06.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345939 -lt 85899345939 2024-04-25T08:58:06.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:06.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953488 2024-04-25T08:58:06.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:06.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:06.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953488 2024-04-25T08:58:06.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:06.040 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953488 2024-04-25T08:58:06.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953488 2024-04-25T08:58:06.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953488' 2024-04-25T08:58:06.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:06.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953488 -lt 137438953488 2024-04-25T08:58:06.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:58:06.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:06.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:07.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:58:07.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:07.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:07.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:07.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:07.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:07.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:07.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:07.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:58:07.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:07.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:07.510 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:08.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:58:08.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:58:08.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:58:08.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-04-25T08:58:08.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:58:08.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-04-25T08:58:08.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-04-25T08:58:08.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-04-25T08:58:08.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:08.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:08.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:08.671 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:08.671 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:58:08.671 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:58:08.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:08.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:08.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:08.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803810 2024-04-25T08:58:08.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803810 2024-04-25T08:58:08.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810' 2024-04-25T08:58:08.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:08.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574877 2024-04-25T08:58:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574877 2024-04-25T08:58:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-55834574877' 2024-04-25T08:58:09.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:09.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:09.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T08:58:09.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T08:58:09.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-55834574877 2-85899345942' 2024-04-25T08:58:09.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:09.150 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:58:09.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953491 2024-04-25T08:58:09.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953491 2024-04-25T08:58:09.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-55834574877 2-85899345942 3-137438953491' 2024-04-25T08:58:09.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:09.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803810 2024-04-25T08:58:09.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:09.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:09.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803810 2024-04-25T08:58:09.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:09.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803810 2024-04-25T08:58:09.295 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803810 2024-04-25T08:58:09.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-04-25T08:58:09.296 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:09.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803810 -lt 25769803810 2024-04-25T08:58:09.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:09.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574877 2024-04-25T08:58:09.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:09.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:09.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:09.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574877 2024-04-25T08:58:09.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574877 2024-04-25T08:58:09.696 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574877 2024-04-25T08:58:09.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-04-25T08:58:09.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:10.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574875 -lt 55834574877 2024-04-25T08:58:10.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:58:11.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:58:11.118 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:11.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574877 -lt 55834574877 2024-04-25T08:58:11.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:11.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345942 2024-04-25T08:58:11.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:11.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:11.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345942 2024-04-25T08:58:11.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:11.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T08:58:11.555 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345942 2024-04-25T08:58:11.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-04-25T08:58:11.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:11.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345942 2024-04-25T08:58:11.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:11.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953491 2024-04-25T08:58:11.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:11.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:11.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953491 2024-04-25T08:58:11.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:11.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953491 2024-04-25T08:58:11.992 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953491 2024-04-25T08:58:11.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953491' 2024-04-25T08:58:11.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:12.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953491 -lt 137438953491 2024-04-25T08:58:12.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T08:58:12.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:58:12.889 INFO:tasks.workunit.client.0.smithi031.stderr:dumped all 2024-04-25T08:58:12.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-04-25T08:58:12.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-04-25T08:58:12.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:12.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:12.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:12.907 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:12.907 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:13.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:13.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 0 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:13.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:13.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:13.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:58:13.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:13.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:58:13.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:13.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:58:13.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:13.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:13.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:13.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:58:13.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:13.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:13.881 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:13.881 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:13.881 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:13.881 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:13.881 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:13.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:13.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:13.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:13.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:58:13.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:13.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:13.996 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:13.997 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:13.997 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:13.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:13.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:13.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 1 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:14.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:14.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:14.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:58:14.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:14.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:14.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:14.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:14.110 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:14.110 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:14.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:58:14.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:14.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:58:14.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:14.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:58:14.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:14.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:14.566 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T08:58:14.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:14.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:14.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:14.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:14.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:58:14.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:58:14.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:58:14.691 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:14.692 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:14.692 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:14.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T08:58:14.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:14.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:14.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:58:14.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:14.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:58:14.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:58:14.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:58:14.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:14.809 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:58:14.809 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:14.809 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:14.810 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:14.810 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:14.810 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:14.810 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:14.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383712 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383712 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:58:14.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-383712 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:14.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:14.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:14.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-04-25T08:58:14.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:58:14.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:58:14.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:58:14.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:58:14.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:58:14.970 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:58:14.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:58:14.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:58:14.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:58:15.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:58:15.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:58:15.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:58:15.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:58:15.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:15.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:15.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:15.533 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:15.533 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:58:15.533 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:58:15.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:15.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:15.533 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803813 2024-04-25T08:58:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803813 2024-04-25T08:58:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813' 2024-04-25T08:58:15.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:15.695 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:15.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574880 2024-04-25T08:58:15.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574880 2024-04-25T08:58:15.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574880' 2024-04-25T08:58:15.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:15.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:15.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345945 2024-04-25T08:58:15.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345945 2024-04-25T08:58:15.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574880 2-85899345945' 2024-04-25T08:58:15.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:15.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:58:16.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953494 2024-04-25T08:58:16.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953494 2024-04-25T08:58:16.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574880 2-85899345945 3-137438953494' 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803813 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803813 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803813 2024-04-25T08:58:16.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-25T08:58:16.153 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803813 2024-04-25T08:58:16.153 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:16.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803811 -lt 25769803813 2024-04-25T08:58:16.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:58:17.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:58:17.600 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:18.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803813 -lt 25769803813 2024-04-25T08:58:18.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:18.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574880 2024-04-25T08:58:18.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:18.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:18.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574880 2024-04-25T08:58:18.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:18.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574880 2024-04-25T08:58:18.030 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574880 2024-04-25T08:58:18.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2024-04-25T08:58:18.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:18.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574880 -lt 55834574880 2024-04-25T08:58:18.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:18.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345945 2024-04-25T08:58:18.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:18.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345945 2024-04-25T08:58:18.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345945 2024-04-25T08:58:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345945' 2024-04-25T08:58:18.500 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345945 2024-04-25T08:58:18.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:19.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345945 -lt 85899345945 2024-04-25T08:58:19.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:19.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953494 2024-04-25T08:58:19.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:19.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:19.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:19.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953494 2024-04-25T08:58:19.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953494 2024-04-25T08:58:19.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953494' 2024-04-25T08:58:19.067 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953494 2024-04-25T08:58:19.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:19.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953495 -lt 137438953494 2024-04-25T08:58:19.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:58:19.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:19.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:20.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:58:20.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:20.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:20.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:20.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:20.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:20.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:20.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:20.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:58:20.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:20.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:20.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:21.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 0 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:21.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:21.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:58:21.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:21.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:21.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:21.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:21.193 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:21.193 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:21.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:21.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:21.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:21.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:21.655 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:58:21.655 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:21.655 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:21.656 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:21.656 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:21.656 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:21.657 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:21.657 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:21.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:21.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:21.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:58:21.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:21.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:21.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:21.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:21.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:21.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:21.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:21.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 2 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:21.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:21.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:21.885 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:21.885 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:22.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:22.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T08:58:22.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:22.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T08:58:22.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:22.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T08:58:22.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:22.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:22.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:22.366 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.0.asok 2024-04-25T08:58:22.367 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:22.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:22.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.0.asok 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:22.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.0.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:22.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383712 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383712 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:58:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-383712 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:22.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:22.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:22.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-04-25T08:58:22.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:58:22.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:58:22.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:58:22.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:58:22.635 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:58:22.635 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:58:22.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:58:22.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:58:22.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:58:22.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:58:22.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:58:22.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:58:22.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:58:22.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:22.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:23.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:23.121 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:23.121 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:58:23.121 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:58:23.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:23.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:23.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:23.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803817 2024-04-25T08:58:23.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803817 2024-04-25T08:58:23.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817' 2024-04-25T08:58:23.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:23.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:23.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574884 2024-04-25T08:58:23.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574884 2024-04-25T08:58:23.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-55834574884' 2024-04-25T08:58:23.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:23.445 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:23.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345949 2024-04-25T08:58:23.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345949 2024-04-25T08:58:23.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-55834574884 2-85899345949' 2024-04-25T08:58:23.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:23.594 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:58:23.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953498 2024-04-25T08:58:23.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953498 2024-04-25T08:58:23.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-55834574884 2-85899345949 3-137438953498' 2024-04-25T08:58:23.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:23.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803817 2024-04-25T08:58:23.737 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:23.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:23.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803817 2024-04-25T08:58:23.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:23.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803817 2024-04-25T08:58:23.740 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803817 2024-04-25T08:58:23.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-04-25T08:58:23.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:24.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803814 -lt 25769803817 2024-04-25T08:58:24.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:58:25.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:58:25.201 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:25.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803817 -lt 25769803817 2024-04-25T08:58:25.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:25.640 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574884 2024-04-25T08:58:25.640 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:25.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:25.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574884 2024-04-25T08:58:25.643 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:25.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574884 2024-04-25T08:58:25.644 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574884 2024-04-25T08:58:25.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574884' 2024-04-25T08:58:25.644 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:26.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574884 -lt 55834574884 2024-04-25T08:58:26.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:26.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:26.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345949 2024-04-25T08:58:26.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:26.062 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345949 2024-04-25T08:58:26.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:26.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345949 2024-04-25T08:58:26.064 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345949 2024-04-25T08:58:26.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345949' 2024-04-25T08:58:26.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:26.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345949 -lt 85899345949 2024-04-25T08:58:26.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:26.494 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953498 2024-04-25T08:58:26.495 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:26.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:26.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953498 2024-04-25T08:58:26.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953498 2024-04-25T08:58:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953498' 2024-04-25T08:58:26.500 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953498 2024-04-25T08:58:26.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:26.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953498 -lt 137438953498 2024-04-25T08:58:26.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:58:26.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:26.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:27.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:58:27.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:27.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:27.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:27.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:27.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:27.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:27.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:27.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:58:27.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:27.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:27.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:28.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:58:28.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:58:28.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:58:28.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-04-25T08:58:28.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:58:28.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-04-25T08:58:28.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-04-25T08:58:28.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-04-25T08:58:28.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:28.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:29.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:29.063 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:29.063 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:58:29.064 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:58:29.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:29.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:29.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:29.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803820 2024-04-25T08:58:29.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803820 2024-04-25T08:58:29.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820' 2024-04-25T08:58:29.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:29.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:29.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574887 2024-04-25T08:58:29.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574887 2024-04-25T08:58:29.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-55834574887' 2024-04-25T08:58:29.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:29.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345952 2024-04-25T08:58:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345952 2024-04-25T08:58:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-55834574887 2-85899345952' 2024-04-25T08:58:29.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:29.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:58:29.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953501 2024-04-25T08:58:29.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953501 2024-04-25T08:58:29.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-55834574887 2-85899345952 3-137438953501' 2024-04-25T08:58:29.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:29.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803820 2024-04-25T08:58:29.669 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:29.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:29.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803820 2024-04-25T08:58:29.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:29.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803820 2024-04-25T08:58:29.672 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803820 2024-04-25T08:58:29.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-04-25T08:58:29.673 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:30.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803818 -lt 25769803820 2024-04-25T08:58:30.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:58:31.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:58:31.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:31.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803820 -lt 25769803820 2024-04-25T08:58:31.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:31.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574887 2024-04-25T08:58:31.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:31.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:31.585 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574887 2024-04-25T08:58:31.586 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:31.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574887 2024-04-25T08:58:31.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574887' 2024-04-25T08:58:31.587 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574887 2024-04-25T08:58:31.588 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574887 -lt 55834574887 2024-04-25T08:58:32.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:32.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345952 2024-04-25T08:58:32.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:32.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:32.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345952 2024-04-25T08:58:32.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:32.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345952 2024-04-25T08:58:32.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345952' 2024-04-25T08:58:32.025 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345952 2024-04-25T08:58:32.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:32.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345952 -lt 85899345952 2024-04-25T08:58:32.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:32.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953501 2024-04-25T08:58:32.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:32.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:32.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953501 2024-04-25T08:58:32.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:32.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953501 2024-04-25T08:58:32.539 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953501 2024-04-25T08:58:32.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953501' 2024-04-25T08:58:32.539 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:32.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953501 -lt 137438953501 2024-04-25T08:58:32.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T08:58:32.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:58:33.387 INFO:tasks.workunit.client.0.smithi031.stderr:dumped all 2024-04-25T08:58:33.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 0 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:58:33.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:33.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:33.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:33.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:33.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:33.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:33.867 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:33.867 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:33.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:33.870 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:33.871 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:33.871 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:33.871 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:33.871 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:33.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:33.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:33.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:33.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T08:58:33.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:33.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 1 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:34.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:34.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:34.109 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:34.109 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:34.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:58:34.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:34.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:58:34.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:34.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:58:34.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:34.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:34.573 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T08:58:34.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:34.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:34.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:34.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:34.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:58:34.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:58:34.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:58:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:34.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T08:58:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:34.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-383712 td/osd-rep-recov-eio.sh 2 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-383712 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T08:58:34.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:34.814 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-383712 2024-04-25T08:58:34.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:58:34.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-383712 2024-04-25T08:58:34.815 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-383712 2024-04-25T08:58:34.815 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:35.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:35.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:35.244 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T08:58:35.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:35.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T08:58:35.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:35.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T08:58:35.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:35.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:35.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:35.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T08:58:35.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:58:35.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:58:35.246 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.0.asok 2024-04-25T08:58:35.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:35.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:35.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:35.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:35.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T08:58:35.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:58:35.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:58:35.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:35.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:35.372 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T08:58:35.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.0.asok 2024-04-25T08:58:35.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:35.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.0.asok injectdataerr pool-rep obj-eio-383712 2024-04-25T08:58:35.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-383712 hang 2024-04-25T08:58:35.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:35.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:58:35.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-383712 2024-04-25T08:58:35.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-04-25T08:58:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-04-25T08:58:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-04-25T08:58:35.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-383712 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:58:40.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-04-25T08:58:40.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-04-25T08:58:40.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-04-25T08:58:40.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:58:40.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:58:40.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:58:40.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:58:40.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:58:40.501 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:58:40.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:58:40.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:58:40.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:58:40.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:58:40.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:58:40.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:58:40.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:58:40.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:40.624 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:41.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:41.054 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T08:58:41.054 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T08:58:41.054 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T08:58:41.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:41.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:41.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:41.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803824 2024-04-25T08:58:41.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803824 2024-04-25T08:58:41.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803824' 2024-04-25T08:58:41.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:41.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:41.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574891 2024-04-25T08:58:41.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574891 2024-04-25T08:58:41.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803824 1-55834574891' 2024-04-25T08:58:41.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:41.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:41.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345956 2024-04-25T08:58:41.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345956 2024-04-25T08:58:41.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-85899345956' 2024-04-25T08:58:41.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:41.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:58:41.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953505 2024-04-25T08:58:41.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953505 2024-04-25T08:58:41.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-85899345956 3-137438953505' 2024-04-25T08:58:41.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:41.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803824 2024-04-25T08:58:41.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:41.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:41.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803824 2024-04-25T08:58:41.666 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:41.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803824 2024-04-25T08:58:41.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-04-25T08:58:41.667 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803824 2024-04-25T08:58:41.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:42.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803822 -lt 25769803824 2024-04-25T08:58:42.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:58:43.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:58:43.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:43.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803824 -lt 25769803824 2024-04-25T08:58:43.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:43.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574891 2024-04-25T08:58:43.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:43.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:43.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574891 2024-04-25T08:58:43.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:43.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574891 2024-04-25T08:58:43.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-04-25T08:58:43.574 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574891 2024-04-25T08:58:43.574 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:44.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574891 -lt 55834574891 2024-04-25T08:58:44.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:44.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345956 2024-04-25T08:58:44.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:44.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:44.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345956 2024-04-25T08:58:44.005 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:44.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345956 2024-04-25T08:58:44.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345956' 2024-04-25T08:58:44.007 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345956 2024-04-25T08:58:44.007 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:44.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345957 -lt 85899345956 2024-04-25T08:58:44.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:44.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953505 2024-04-25T08:58:44.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:44.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:58:44.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953505 2024-04-25T08:58:44.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:44.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953505 2024-04-25T08:58:44.469 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953505 2024-04-25T08:58:44.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953505' 2024-04-25T08:58:44.470 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:58:44.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953506 -lt 137438953505 2024-04-25T08:58:44.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:58:44.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:44.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:45.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:58:45.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:45.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:45.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:45.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:45.474 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:45.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:45.475 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:45.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:45.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:45.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:45.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:46.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T08:58:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:58:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T08:58:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:46.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:46.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:46.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:46.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:46.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:46.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:46.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:46.554 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:46.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:46.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:46.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:46.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:47.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:47.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:48.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:58:48.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:58:48.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T08:58:48.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:48.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T08:58:48.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:48.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:48.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:48.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:48.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:48.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:48.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:48.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:48.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:48.781 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:48.781 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:48.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:49.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:49.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:58:49.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:49.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:49.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:49.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:49.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:49.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:49.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:49.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:58:49.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:58:49.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-25T08:58:49.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:49.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T08:58:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:50.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:50.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:50.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:50.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:50.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:50.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:50.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:50.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:50.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:50.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:50.762 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:51.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:51.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:51.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:51.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:58:51.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:58:51.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-25T08:58:51.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:51.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T08:58:52.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:52.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:52.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:52.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:52.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:52.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:52.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:52.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:53.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:53.107 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:53.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:53.108 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:53.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:53.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:58:53.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:53.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:53.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:53.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:53.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:53.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:53.668 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:54.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:58:54.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:58:54.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-04-25T08:58:54.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:54.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T08:58:55.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:55.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:55.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:55.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:55.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:55.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:55.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:56.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:56.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:56.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:56.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:56.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:56.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:58:56.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:56.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:56.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:56.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:56.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:56.844 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:56.844 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:58:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:58:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-04-25T08:58:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T08:59:00.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:59:00.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:59:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:59:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:59:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:59:00.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:59:00.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:59:00.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:59:01.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:59:01.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:59:01.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:59:01.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:59:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:59:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:59:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:59:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:59:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:59:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:59:01.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:59:01.600 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:59:01.600 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:59:02.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:59:02.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:59:02.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-04-25T08:59:02.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:59:02.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T08:59:08.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:59:08.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:59:08.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:59:08.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:59:08.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:59:08.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:59:08.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:59:08.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:59:08.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:59:08.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:59:08.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:59:08.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:59:09.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:59:09.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:59:09.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:59:10.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:59:10.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:59:10.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-04-25T08:59:10.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:59:10.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-04-25T08:59:22.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:59:22.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:59:22.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:59:22.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:59:22.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:59:22.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:59:22.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:59:22.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:59:23.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:59:23.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:59:23.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:59:23.370 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:59:23.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:59:23.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:59:23.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:59:24.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:59:24.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:59:24.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-04-25T08:59:24.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:59:24.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T08:59:39.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:59:39.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:59:39.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:59:39.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:59:39.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:59:39.472 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:59:39.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:59:39.473 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:59:39.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:59:39.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:59:39.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:59:39.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:59:40.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:59:40.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:59:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:59:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:59:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:59:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:59:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:59:40.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:59:40.462 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:59:41.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:59:41.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:59:41.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-04-25T08:59:41.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:59:41.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T08:59:56.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:59:56.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:59:56.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:59:56.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:59:56.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:59:56.046 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:59:56.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:59:56.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:59:56.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:59:56.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:59:56.477 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:59:56.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:59:57.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:59:57.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:59:57.024 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:59:57.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:59:57.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:59:57.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-04-25T08:59:57.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:59:57.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:00:12.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:00:12.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:12.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:12.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:12.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:12.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:12.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:12.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:12.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:00:12.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:12.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:12.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:13.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:00:13.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:00:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:00:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:00:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-04-25T09:00:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:00:14.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:00:29.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:00:29.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:29.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:29.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:29.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:29.147 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:29.148 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:29.148 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:29.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:00:29.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:29.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:29.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:30.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:00:30.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:00:30.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:00:30.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:00:30.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:00:30.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:00:30.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:00:30.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:00:30.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:00:30.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:00:30.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:00:30.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-04-25T09:00:30.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:00:30.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-04-25T09:00:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:00:35.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:35.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:35.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:35.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:35.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:35.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:35.219 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:35.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:00:35.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:35.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:35.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:36.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:00:36.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:00:36.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:00:36.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:00:36.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:00:36.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-04-25T09:00:36.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T09:00:37.281 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T09:00:37.281 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_fingerprint": "159a6fb4-30d7-498e-996f-d843ba902f08", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-04-25T09:00:37.290448+0000", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "tag": "", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "health": { 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "status": "HEALTH_ERR", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "checks": { 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "OBJECT_UNFOUND": { 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T09:00:37.282 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1/1 objects unfound (100.000%)", 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DAMAGED": { 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_ERR", 2024-04-25T09:00:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "PG_DEGRADED": { 2024-04-25T09:00:37.284 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T09:00:37.285 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "2 pool(s) do not have an application enabled", 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 2 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [ 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'foo'" 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.286 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "SLOW_OPS": { 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "summary": { 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "message": "1 slow ops, oldest one blocked for 116 sec, osd.3 has slow ops", 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: "count": 1 2024-04-25T09:00:37.287 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "detail": [], 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "muted": false 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "mutes": [] 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_first_committed": 1, 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap_last_committed": 1, 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "monmap": { 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "2bb1c137-dcba-43b0-a004-7c88fee34ade", 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T08:55:28.312244Z", 2024-04-25T09:00:37.288 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T08:55:28.312244Z", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release": 19, 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "min_mon_release_name": "squid", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "election_strategy": 1, 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "disallowed_leaders: ": "", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": false, 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "tiebreaker_mon": "", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_ranks: ": "", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "features": { 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "persistent": [ 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "kraken", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "luminous", 2024-04-25T09:00:37.289 INFO:tasks.workunit.client.0.smithi031.stdout: "mimic", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap-prune", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "nautilus", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "octopus", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "pacific", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "elector-pinging", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "quincy", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "reef", 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "squid" 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: "optional": [] 2024-04-25T09:00:37.290 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "mons": [ 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "rank": 0, 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "a", 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7140", 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 0 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.291 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:7140/0", 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:7140/0", 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: "priority": 0, 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 0, 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_location": "{}" 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: "quorum": [ 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: 0 2024-04-25T09:00:37.292 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap": { 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 38, 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "fsid": "2bb1c137-dcba-43b0-a004-7c88fee34ade", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "created": "2024-04-25T08:55:29.270008+0000", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T08:57:56.353657+0000", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "last_up_change": "2024-04-25T08:57:46.928673+0000", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "last_in_change": "2024-04-25T08:57:12.566890+0000", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_num": 5799936, 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_set": [ 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "pglog_hardlimit", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snapdirs", 2024-04-25T09:00:37.293 INFO:tasks.workunit.client.0.smithi031.stdout: "recovery_deletes", 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "sortbitwise" 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_version": 9, 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_snapshot": "", 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_max": 2, 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "max_osd": 4, 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "require_min_compat_client": "luminous", 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "min_compat_client": "luminous", 2024-04-25T09:00:37.294 INFO:tasks.workunit.client.0.smithi031.stdout: "require_osd_release": "squid", 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "allow_crimson": false, 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "pools": [ 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "foo", 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T08:55:30.402775+0000", 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T09:00:37.295 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 8, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 8, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 8, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 8, 2024-04-25T09:00:37.296 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 8, 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "9", 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T09:00:37.297 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T09:00:37.298 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T09:00:37.299 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T09:00:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 1.5, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 1.5, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 1, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 1.5, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 1.5, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.301 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 2, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_name": "pool-rep", 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "create_time": "2024-04-25T08:57:54.890265+0000", 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "flags": 1, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "flags_names": "hashpspool", 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "size": 3, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "min_size": 2, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_rule": 0, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-04-25T09:00:37.302 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "object_hash": 2, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_autoscale_mode": "off", 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num": 1, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num": 1, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_placement_num_target": 1, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_target": 1, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_num_pending": 1, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pg_merge_meta": { 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "source_pgid": "0.0", 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "ready_epoch": 0, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_started": 0, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": 0, 2024-04-25T09:00:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: "source_version": "0'0", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "target_version": "0'0" 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "last_change": "38", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend": "0", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "auid": 0, 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_mode": "selfmanaged", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_seq": 0, 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_epoch": 0, 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_snaps": [], 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps": "[]", 2024-04-25T09:00:37.304 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_bytes": 0, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "quota_max_objects": 0, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "tiers": [], 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "tier_of": -1, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "read_tier": -1, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "write_tier": -1, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_mode": "none", 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_bytes": 0, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "target_max_objects": 0, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T09:00:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_flush_age": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "cache_min_evict_age": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profile": "", 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_params": { 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "none" 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_period": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_count": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "use_gmt_hitset": true, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T09:00:37.306 INFO:tasks.workunit.client.0.smithi031.stdout: "hit_set_search_last_n": 0, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "grade_table": [], 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "stripe_width": 0, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "expected_num_objects": 0, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "fast_read": false, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "options": {}, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "application_metadata": {}, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "read_balance": { 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "score_type": "Fair distribution", 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "score_acting": 4, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "score_stable": 4, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_score": 0.75, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_acting": 3, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "raw_score_stable": 3, 2024-04-25T09:00:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity_weighted": 1, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity": 1, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "osds": [ 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "0340ee48-d1f5-4995-a505-2e681da5c4ff", 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T09:00:37.308 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 6, 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 34, 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 0, 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6802", 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.309 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6803", 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6804", 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6805", 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.311 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6808", 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6809", 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.312 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6806", 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6807", 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1003314867 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6803/1003314867", 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6805/1003314867", 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1003314867", 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1003314867", 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "973ea054-a144-451c-81cf-4f6dc7bfe378", 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T09:00:37.314 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 13, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 34, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 0, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6810", 2024-04-25T09:00:37.315 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6811", 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.316 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6812", 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6813", 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6816", 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6817", 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6814", 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6815", 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 4172694295 2024-04-25T09:00:37.319 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6811/4172694295", 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6813/4172694295", 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4172694295", 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4172694295", 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.320 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "15c56d19-20e8-4cc2-af3a-68ab26975bee", 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 20, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 34, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 0, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T09:00:37.321 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6818", 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6819", 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6820", 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6821", 2024-04-25T09:00:37.323 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6824", 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.324 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6825", 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6822", 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.325 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6823", 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 1355598743 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6819/1355598743", 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6821/1355598743", 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6825/1355598743", 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6823/1355598743", 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T09:00:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 3, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "uuid": "04121504-b0eb-4008-8fc1-8c636c36c097", 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 1, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "in": 1, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 1, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_affinity": 1, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_begin": 0, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "last_clean_end": 0, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 32, 2024-04-25T09:00:37.327 INFO:tasks.workunit.client.0.smithi031.stdout: "up_thru": 36, 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "down_at": 0, 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "lost_at": 0, 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addrs": { 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6826", 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.328 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6827", 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addrs": { 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6828", 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6829", 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addrs": { 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6832", 2024-04-25T09:00:37.330 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6833", 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addrs": { 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: "addrvec": [ 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v2", 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6830", 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "v1", 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "addr": "127.0.0.1:6831", 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "nonce": 3863315329 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "public_addr": "127.0.0.1:6827/3863315329", 2024-04-25T09:00:37.332 INFO:tasks.workunit.client.0.smithi031.stdout: "cluster_addr": "127.0.0.1:6829/3863315329", 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_back_addr": "127.0.0.1:6833/3863315329", 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "heartbeat_front_addr": "127.0.0.1:6831/3863315329", 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "state": [ 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "exists", 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "up" 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_xinfo": [ 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "0.000000", 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T08:56:02.026938+0000", 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.334 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "0.000000", 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T08:56:36.897050+0000", 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.335 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "0.000000", 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T08:57:11.385776+0000", 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 3, 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "down_stamp": "0.000000", 2024-04-25T09:00:37.336 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_probability": 0, 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "laggy_interval": 0, 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "features": 4540701547738038271, 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "old_weight": 0, 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "last_purged_snaps_scrub": "2024-04-25T08:57:46.583698+0000", 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "dead_epoch": 0 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap": [], 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_items": [ 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.337 INFO:tasks.workunit.client.0.smithi031.stdout: "pgid": "1.0", 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "mappings": [ 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "from": 0, 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "to": 3 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_upmap_primaries": [], 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "pg_temp": [], 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "primary_temp": [], 2024-04-25T09:00:37.338 INFO:tasks.workunit.client.0.smithi031.stdout: "blocklist": {}, 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "range_blocklist": {}, 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "erasure_code_profiles": { 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "default": { 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "k": "2", 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "m": "2", 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "plugin": "jerasure", 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "technique": "reed_sol_van" 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "removed_snaps_queue": [], 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "new_removed_snaps": [], 2024-04-25T09:00:37.339 INFO:tasks.workunit.client.0.smithi031.stdout: "new_purged_snaps": [], 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "crush_node_flags": {}, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "device_class_flags": {}, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode": { 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_enabled": false, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_bucket_count": 0, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "degraded_stretch_mode": 0, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "recovering_stretch_mode": 0, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "stretch_mode_bucket": 0 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_metadata": [ 2024-04-25T09:00:37.340 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6804/1003314867,v1:127.0.0.1:6805/1003314867]", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:00:37.341 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:00:37.342 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T08:55:36.121008Z", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6802/1003314867,v1:127.0.0.1:6803/1003314867]", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T09:00:37.343 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1003314867,v1:127.0.0.1:6809/1003314867]", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1003314867,v1:127.0.0.1:6807/1003314867]", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-04-25T09:00:37.344 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6812/4172694295,v1:127.0.0.1:6813/4172694295]", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:00:37.345 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T09:00:37.346 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T08:56:09.406302Z", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T09:00:37.347 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6810/4172694295,v1:127.0.0.1:6811/4172694295]", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4172694295,v1:127.0.0.1:6817/4172694295]", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4172694295,v1:127.0.0.1:6815/4172694295]", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T09:00:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6820/1355598743,v1:127.0.0.1:6821/1355598743]", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T09:00:37.349 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:00:37.350 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T08:56:44.469423Z", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T09:00:37.351 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6818/1355598743,v1:127.0.0.1:6819/1355598743]", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/1355598743,v1:127.0.0.1:6825/1355598743]", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/1355598743,v1:127.0.0.1:6823/1355598743]", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T09:00:37.352 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 3, 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "arch": "x86_64", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "back_addr": "[v2:127.0.0.1:6828/3863315329,v1:127.0.0.1:6829/3863315329]", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "back_iface": "", 2024-04-25T09:00:37.353 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs": "1", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:00:37.354 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_release": "squid", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "created_at": "2024-04-25T08:57:18.795053Z", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "default_device_class": "hdd", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQEW", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:00:37.355 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": "sda", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "distro": "ubuntu", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "distro_version": "22.04", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "front_addr": "[v2:127.0.0.1:6826/3863315329,v1:127.0.0.1:6827/3863315329]", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "front_iface": "", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/3863315329,v1:127.0.0.1:6833/3863315329]", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/3863315329,v1:127.0.0.1:6831/3863315329]", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "hostname": "smithi031", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "journal_rotational": "1", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:00:37.356 INFO:tasks.workunit.client.0.smithi031.stdout: "mem_total_kb": "32753056", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "os": "Linux", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "osdspec_affinity": "", 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "rotational": "1" 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_clean_epochs": { 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "min_last_epoch_clean": 0, 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "last_epoch_clean": { 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: "per_pool": [] 2024-04-25T09:00:37.357 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_epochs": [ 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 6 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 13 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.358 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 20 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 3, 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 34 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_first_committed": 1, 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: "osdmap_last_committed": 38, 2024-04-25T09:00:37.359 INFO:tasks.workunit.client.0.smithi031.stdout: "crushmap": { 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "devices": [ 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.0", 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.1", 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T09:00:37.360 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.2", 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 3, 2024-04-25T09:00:37.361 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd.3", 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: "class": "hdd" 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: "types": [ 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 0, 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "osd" 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T09:00:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "host" 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 2, 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "chassis" 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 3, 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "rack" 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 4, 2024-04-25T09:00:37.363 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "row" 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 5, 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pdu" 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 6, 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "pod" 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 7, 2024-04-25T09:00:37.364 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "room" 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 8, 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "datacenter" 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 9, 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "zone" 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 10, 2024-04-25T09:00:37.365 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "region" 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "root" 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "buckets": [ 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -1, 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default", 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T09:00:37.366 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 25608, 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 25608, 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.367 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -2, 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "default~hdd", 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 11, 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "root", 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 25608, 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 25608, 2024-04-25T09:00:37.368 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -3, 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031", 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 25608, 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T09:00:37.369 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.370 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 3, 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 3 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.371 INFO:tasks.workunit.client.0.smithi031.stdout: "id": -4, 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "name": "smithi031~hdd", 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "type_id": 1, 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "type_name": "host", 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 25608, 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "alg": "straw2", 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "hash": "rjenkins1", 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "items": [ 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 0, 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 0 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.372 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 1, 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 1 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 2, 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 2 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.373 INFO:tasks.workunit.client.0.smithi031.stdout: "id": 3, 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "weight": 6402, 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "pos": 3 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "rules": [ 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_id": 0, 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "rule_name": "replicated_rule", 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "type": 1, 2024-04-25T09:00:37.374 INFO:tasks.workunit.client.0.smithi031.stdout: "steps": [ 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "take", 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "item": -1, 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "item_name": "default" 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "choose_firstn", 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 0, 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "type": "osd" 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.375 INFO:tasks.workunit.client.0.smithi031.stdout: "op": "emit" 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: ] 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "tunables": { 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_tries": 0, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_local_fallback_tries": 0, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_total_tries": 50, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_descend_once": 1, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_vary_r": 1, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "chooseleaf_stable": 1, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_descents": 100, 2024-04-25T09:00:37.376 INFO:tasks.workunit.client.0.smithi031.stdout: "msr_collision_tries": 100, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "straw_calc_version": 1, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "allowed_bucket_algs": 54, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "profile": "jewel", 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "optimal_tunables": 1, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "legacy_tunables": 0, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "minimum_required_version": "jewel", 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables": 1, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables2": 1, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v2_rules": 0, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables3": 1, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v3_rules": 0, 2024-04-25T09:00:37.377 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v4_buckets": 1, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "require_feature_tunables5": 1, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "has_v5_rules": 0, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "has_msr_rules": 0 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "choose_args": {} 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "fsmap": { 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 1, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "default_fscid": -1, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": { 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "compat": {}, 2024-04-25T09:00:37.378 INFO:tasks.workunit.client.0.smithi031.stdout: "ro_compat": {}, 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "incompat": { 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_1": "base v0.20", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_2": "client writeable ranges", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_4": "dir inode in separate object", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_8": "no anchor table", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_9": "file layout v2", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_10": "snaprealm v2", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_11": "minor log segments", 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T09:00:37.379 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "feature_flags": { 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "enable_multiple": true, 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "ever_enabled_multiple": true 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "standbys": [], 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "filesystems": [] 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_first_committed": 1, 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "mdsmap_last_committed": 1, 2024-04-25T09:00:37.380 INFO:tasks.workunit.client.0.smithi031.stdout: "auth": { 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 1, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 6, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_secrets": 4 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 9, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_active": 9, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_unknown": 0, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osd": 4, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_sum": { 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 4096, 2024-04-25T09:00:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 1, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 3, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 3, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 1, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 1, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 3, 2024-04-25T09:00:37.382 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 12, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 1, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 4, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 3, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 12288, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T09:00:37.383 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T09:00:37.384 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 3 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T09:00:37.385 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 1, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 1, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 27, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 27, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 0 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum": { 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T09:00:37.386 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 27, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 4, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 4, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 419430400, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 108572, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 1260, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 6, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 107257, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 419321828, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 429496729600, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 429385551872, 2024-04-25T09:00:37.387 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1290240, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 435516, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 6378, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 109831958 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T09:00:37.388 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 4, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [ 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.389 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: 1 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 262144 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.390 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_sum_by_class": { 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "hdd": { 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "up_from": 0, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 0, 2024-04-25T09:00:37.391 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pgs": 27, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "num_osds": 4, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_osds": 4, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "kb": 419430400, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used": 108572, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_data": 1260, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_omap": 6, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_used_meta": 107257, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "kb_avail": 419321828, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "statfs": { 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 429496729600, 2024-04-25T09:00:37.392 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 429385551872, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 1290240, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 435516, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 6378, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 109831958 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "hb_peers": [], 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_trim_queue_len": 0, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "num_snap_trimming": 0, 2024-04-25T09:00:37.393 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shards_repaired": 4, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: "op_queue_age_hist": { 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: "histogram": [ 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.394 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: 1 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: "upper_bound": 262144 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: "perf_stat": { 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ms": 0, 2024-04-25T09:00:37.395 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ms": 0, 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "commit_latency_ns": 0, 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "apply_latency_ns": 0 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "alerts": [], 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "network_ping_times": [] 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "pool_stats": [ 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 2, 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 1, 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T09:00:37.396 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 4096, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 1, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 3, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 3, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 1, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 1, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 3, 2024-04-25T09:00:37.397 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 12, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 1, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 4, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 3, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 12288, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:00:37.398 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T09:00:37.399 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 3 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 12288, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 12288, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T09:00:37.400 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 1, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 1, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 3, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 3, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 3 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "poolid": 1, 2024-04-25T09:00:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg": 8, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "stat_sum": { 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_clones": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_object_copies": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_missing": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_degraded": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_misplaced": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_unfound": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_dirty": 0, 2024-04-25T09:00:37.402 INFO:tasks.workunit.client.0.smithi031.stdout: "num_whiteouts": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_read_kb": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_write_kb": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_scrub_errors": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_recovered": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_recovered": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_keys_recovered": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_omap": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:00:37.403 INFO:tasks.workunit.client.0.smithi031.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_kb": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_kb": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_promote": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_high": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_flush_mode_low": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_some": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_evict_mode_full": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_pinned": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:00:37.404 INFO:tasks.workunit.client.0.smithi031.stdout: "num_large_omap_objects": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_manifest": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_bytes": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "num_omap_keys": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "num_objects_repaired": 0 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "store_stats": { 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "total": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "available": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "internally_reserved": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "allocated": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "data_stored": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed": 0, 2024-04-25T09:00:37.405 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_allocated": 0, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "data_compressed_original": 0, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "omap_allocated": 0, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "internal_metadata": 0 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "log_size": 0, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "ondisk_log_size": 0, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "up": 24, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "acting": 24, 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: "num_store_stats": 4 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.406 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "osd_stats": [ 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 25769803847 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 55834574914 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 85899345979 2024-04-25T09:00:37.407 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 3, 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "seq": 137438953528 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_state": [ 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+recovery_unfound+degraded", 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 1 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "state": "active+clean", 2024-04-25T09:00:37.408 INFO:tasks.workunit.client.0.smithi031.stdout: "num": 8 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "num_pg_by_osd": [ 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 3, 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 4, 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 8, 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 2, 2024-04-25T09:00:37.409 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 5, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 1, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 3, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 6, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "osd": 0, 2024-04-25T09:00:37.410 INFO:tasks.workunit.client.0.smithi031.stdout: "num_primary_pg": 1, 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "num_acting_pg": 8, 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": [ 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 1, 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: { 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "pool": 2, 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: "purged_snaps": {} 2024-04-25T09:00:37.411 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: ], 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "servicemap": { 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "epoch": 6, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "modified": "2024-04-25T08:58:48.985333+0000", 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "services": {} 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: }, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_first_committed": 1, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "mgrstat_last_committed": 143, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_first_committed": 1, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "logm_last_committed": 599, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "paxos": { 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "first_committed": 252, 2024-04-25T09:00:37.412 INFO:tasks.workunit.client.0.smithi031.stdout: "last_committed": 784, 2024-04-25T09:00:37.413 INFO:tasks.workunit.client.0.smithi031.stdout: "last_pn": 0, 2024-04-25T09:00:37.413 INFO:tasks.workunit.client.0.smithi031.stdout: "accepted_pn": 0 2024-04-25T09:00:37.413 INFO:tasks.workunit.client.0.smithi031.stdout: } 2024-04-25T09:00:37.413 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T09:00:37.414 INFO:tasks.workunit.client.0.smithi031.stderr:report 1473757607 2024-04-25T09:00:37.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T09:00:37.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:00:37.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-04-25T09:00:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-04-25T09:00:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-04-25T09:00:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-04-25T09:00:37.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:00:37.456 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:37.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:00:38.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803849 2024-04-25T09:00:38.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803849 2024-04-25T09:00:38.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803849' 2024-04-25T09:00:38.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:38.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:00:38.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574916 2024-04-25T09:00:38.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574916 2024-04-25T09:00:38.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803849 1-55834574916' 2024-04-25T09:00:38.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:38.156 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:00:38.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345982 2024-04-25T09:00:38.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345982 2024-04-25T09:00:38.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803849 1-55834574916 2-85899345982' 2024-04-25T09:00:38.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:38.299 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:00:38.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953531 2024-04-25T09:00:38.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953531 2024-04-25T09:00:38.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803849 1-55834574916 2-85899345982 3-137438953531' 2024-04-25T09:00:38.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:38.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803849 2024-04-25T09:00:38.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:38.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:00:38.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803849 2024-04-25T09:00:38.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:38.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803849 2024-04-25T09:00:38.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803849' 2024-04-25T09:00:38.493 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803849 2024-04-25T09:00:38.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:00:38.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803847 -lt 25769803849 2024-04-25T09:00:38.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:00:39.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:00:39.895 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:00:40.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803850 -lt 25769803849 2024-04-25T09:00:40.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:40.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574916 2024-04-25T09:00:40.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:40.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:00:40.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574916 2024-04-25T09:00:40.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:40.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574916 2024-04-25T09:00:40.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574916' 2024-04-25T09:00:40.353 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574916 2024-04-25T09:00:40.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:00:40.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574917 -lt 55834574916 2024-04-25T09:00:40.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:40.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345982 2024-04-25T09:00:40.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:40.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:00:40.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345982 2024-04-25T09:00:40.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:40.800 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345982 2024-04-25T09:00:40.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345982 2024-04-25T09:00:40.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345982' 2024-04-25T09:00:40.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:00:41.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345982 -lt 85899345982 2024-04-25T09:00:41.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:41.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953531 2024-04-25T09:00:41.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:41.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:00:41.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:41.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953531 2024-04-25T09:00:41.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953531 2024-04-25T09:00:41.226 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953531 2024-04-25T09:00:41.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953531' 2024-04-25T09:00:41.227 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:00:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953531 -lt 137438953531 2024-04-25T09:00:41.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:00:41.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:00:42.110 INFO:tasks.workunit.client.0.smithi031.stderr:dumped all 2024-04-25T09:00:42.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-04-25T09:00:42.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-04-25T09:00:42.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-04-25T09:00:42.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T09:00:42.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T09:00:42.752 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'pool-rep' does not exist 2024-04-25T09:00:42.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:00:42.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:42.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:00:42.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:00:42.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:00:42.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:00:42.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:00:42.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:00:42.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:00:43.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:00:43.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:00:43.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:00:43.083 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:00:43.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:00:43.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:00:43.085 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:00:43.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:00:43.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:00:43.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:00:43.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:00:43.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:00:43.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:00:43.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:00:43.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:43.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:43.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:00:43.125 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:00:43.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:00:43.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:00:43.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:00:43.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:00:43.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:00:43.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:00:43.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:00:43.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:00:43.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:00:43.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:00:43.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:00:43.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:00:43.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:00:43.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:00:43.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:00:43.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:00:43.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:00:43.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:00:43.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:43.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:43.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:00:43.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:00:43.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:00:43.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:00:43.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:43.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:43.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383712 2024-04-25T09:00:43.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:00:43.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:00:43.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:00:43.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:00:43.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:00:43.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:00:43.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:43.614 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:43.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:00:43.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:00:43.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:00:43.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:00:43.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:00:43.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:00:43.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:00:43.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:00:43.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:00:43.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:00:43.834 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:00:43.834 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:00:43.834 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:43.835 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:43.835 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T09:00:43.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:00:43.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get fsid 2024-04-25T09:00:43.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:00:43.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:00:43.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:00:43.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:00:43.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T09:00:43.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:00:43.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get mon_host 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:00:44.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:00:44.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:00:44.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:00:44.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:00:44.996 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:00:44.996 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:44.996 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:44.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:00:44.997 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:00:44.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:00:45.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:00:45.311 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T09:00:45.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-04-25T09:00:45.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:45.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-04-25T09:00:45.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-04-25T09:00:45.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-04-25T09:00:45.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T09:00:45.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:00:45.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:00:45.331 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T09:00:45.332 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:00:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:00:45.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:00:45.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:00:45.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:00:45.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5ae3a340-94da-4d1c-8b03-42c8a98f998d 2024-04-25T09:00:45.339 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 5ae3a340-94da-4d1c-8b03-42c8a98f998d 2024-04-25T09:00:45.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5ae3a340-94da-4d1c-8b03-42c8a98f998d' 2024-04-25T09:00:45.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:00:45.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9GypmaW43FhAATsmJvivePk0W94Qx6HiZfg== 2024-04-25T09:00:45.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9GypmaW43FhAATsmJvivePk0W94Qx6HiZfg=="}' 2024-04-25T09:00:45.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5ae3a340-94da-4d1c-8b03-42c8a98f998d -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:00:45.712 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:00:45.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:00:45.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9GypmaW43FhAATsmJvivePk0W94Qx6HiZfg== --osd-uuid 5ae3a340-94da-4d1c-8b03-42c8a98f998d 2024-04-25T09:00:45.768 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.779+0000 7f98aecc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:45.781 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.791+0000 7f98aecc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:45.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.791+0000 7f98aecc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:45.782 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.791+0000 7f98aecc97c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:00:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.791+0000 7f98aecc97c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:00:45.783 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.791+0000 7f98aecc97c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:00:45.821 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:45.831+0000 7f98aecc97c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:00:51.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:00:51.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:00:51.701 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:00:51.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:00:51.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:00:52.289 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:00:52.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:00:52.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:00:52.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:00:52.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:00:52.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:00:52.321 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:52.331+0000 7fbb7d1447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:52.336 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:52.347+0000 7fbb7d1447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:52.338 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:52.347+0000 7fbb7d1447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:52.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:00:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:00:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:00:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:00:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:00:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:52.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:00:52.771 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:00:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:00:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:52.913 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:52.923+0000 7fbb7d1447c0 -1 Falling back to public interface 2024-04-25T09:00:53.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:54.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:54.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:54.207 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:00:54.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:00:54.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:54.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:00:54.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:55.305 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:00:55.315+0000 7fbb7d1447c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:00:55.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:55.648 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:00:55.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:55.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:00:55.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:55.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:00:56.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:57.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:57.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:57.314 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:00:57.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:00:57.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:57.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:00:57.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:58.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:58.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:58.753 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:00:58.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:00:58.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:58.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:00:59.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:00.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:00.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:00.227 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:01:00.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:01:00.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:00.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:00.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:01.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:01.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:01.676 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:01:01.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:01:01.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:01.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:02.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:03.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:03.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:03.184 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:01:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:01:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:03.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:03.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:04.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:04.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:04.613 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:01:04.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:01:04.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:04.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:05.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:06.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:06.157 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:01:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:01:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:06.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:06.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:07.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:07.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:07.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:01:07.584 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:01:07.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:07.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:08.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:09.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:09.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:09.335 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:01:09.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:01:09.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:09.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:09.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:10.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:10.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:10.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:01:10.975 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:01:10.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:10.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:11.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:12.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:12.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:12.988 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:01:12.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:01:12.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:12.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:13.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:14.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:14.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:14.896 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:01:14.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:01:14.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:14.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:15.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:16.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:16.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:16.687 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:01:16.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:01:16.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:16.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:17.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:18.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:01:18.301 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:01:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:18.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:18.746 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1872853370,v1:127.0.0.1:6803/1872853370] [v2:127.0.0.1:6804/1872853370,v1:127.0.0.1:6805/1872853370] exists,up 5ae3a340-94da-4d1c-8b03-42c8a98f998d 2024-04-25T09:01:18.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:01:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:01:18.751 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:01:18.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:01:18.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:01:18.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:01:18.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:01:18.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:01:18.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d855eb54-cdba-454e-b2e9-f9fef8ea2f51 2024-04-25T09:01:18.754 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 d855eb54-cdba-454e-b2e9-f9fef8ea2f51 2024-04-25T09:01:18.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d855eb54-cdba-454e-b2e9-f9fef8ea2f51' 2024-04-25T09:01:18.754 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:01:18.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeGypmgXHQLhAAX6KGjG1U+7yHTueufpeLog== 2024-04-25T09:01:18.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeGypmgXHQLhAAX6KGjG1U+7yHTueufpeLog=="}' 2024-04-25T09:01:18.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d855eb54-cdba-454e-b2e9-f9fef8ea2f51 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:01:19.356 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:01:19.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:01:19.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeGypmgXHQLhAAX6KGjG1U+7yHTueufpeLog== --osd-uuid d855eb54-cdba-454e-b2e9-f9fef8ea2f51 2024-04-25T09:01:19.411 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.419+0000 7ffa1baaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:19.437 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.447+0000 7ffa1baaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:19.438 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.447+0000 7ffa1baaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:19.439 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.447+0000 7ffa1baaf7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:01:19.439 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.447+0000 7ffa1baaf7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:01:19.439 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.447+0000 7ffa1baaf7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:01:19.528 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:19.539+0000 7ffa1baaf7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:01:25.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:01:25.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:01:25.563 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:01:25.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:01:25.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:01:26.144 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:01:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:01:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:01:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:01:26.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:01:26.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:01:26.169 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:26.179+0000 7f79056707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:26.171 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:26.183+0000 7f79056707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:26.173 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:26.183+0000 7f79056707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:26.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:01:26.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:01:26.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:01:26.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:01:26.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:01:26.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:26.607 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:01:26.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:01:26.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:26.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:26.732 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:26.743+0000 7f79056707c0 -1 Falling back to public interface 2024-04-25T09:01:27.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:28.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:28.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:28.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:01:28.109 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:01:28.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:28.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:28.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:29.235 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:29.243+0000 7f79056707c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:01:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:29.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:29.586 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:01:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:01:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:29.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:30.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:31.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:31.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:31.333 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:01:31.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:01:31.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:31.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:31.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:32.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:32.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:32.781 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:01:32.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:01:32.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:32.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:33.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:34.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:34.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:34.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:01:34.321 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:01:34.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:34.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:34.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:35.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:35.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:35.777 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:01:35.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:01:35.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:35.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:36.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:37.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:37.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:37.406 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:01:37.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:01:37.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:37.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:37.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:38.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:38.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:38.827 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:01:38.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:01:38.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:38.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:39.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:40.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:40.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:40.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:01:40.303 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:01:40.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:40.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:40.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:41.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:41.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:41.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:01:41.939 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:01:41.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:41.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:42.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:43.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:43.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:43.533 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:01:43.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:01:43.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:43.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:44.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:45.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:45.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:45.823 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:01:45.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:01:45.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:45.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:46.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:47.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:47.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:47.919 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:01:47.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:01:47.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:47.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:48.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:49.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:49.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:49.359 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:01:49.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:01:49.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:49.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:50.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:51.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:51.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:51.095 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:01:51.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:01:51.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:51.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:51.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:52.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:52.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:52.642 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:01:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:01:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:52.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:01:53.076 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3518061743,v1:127.0.0.1:6811/3518061743] [v2:127.0.0.1:6812/3518061743,v1:127.0.0.1:6813/3518061743] exists,up d855eb54-cdba-454e-b2e9-f9fef8ea2f51 2024-04-25T09:01:53.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:01:53.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:01:53.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:53.081 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:01:53.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:01:53.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:01:53.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:01:53.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:01:53.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:01:53.083 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:01:53.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c17a7836-022c-4706-a0d6-153f9764dcfa 2024-04-25T09:01:53.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c17a7836-022c-4706-a0d6-153f9764dcfa' 2024-04-25T09:01:53.084 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 c17a7836-022c-4706-a0d6-153f9764dcfa 2024-04-25T09:01:53.084 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:01:53.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABHCpm8wTXBhAA+FlaXQ38qtH/OXtt/Qih7w== 2024-04-25T09:01:53.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABHCpm8wTXBhAA+FlaXQ38qtH/OXtt/Qih7w=="}' 2024-04-25T09:01:53.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c17a7836-022c-4706-a0d6-153f9764dcfa -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:01:53.658 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:01:53.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:01:53.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABHCpm8wTXBhAA+FlaXQ38qtH/OXtt/Qih7w== --osd-uuid c17a7836-022c-4706-a0d6-153f9764dcfa 2024-04-25T09:01:53.711 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.718+0000 7f4b834647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:53.713 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.722+0000 7f4b834647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:53.715 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.726+0000 7f4b834647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:53.715 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.726+0000 7f4b834647c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:01:53.715 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.726+0000 7f4b834647c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:01:53.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.726+0000 7f4b834647c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:01:53.779 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:01:53.790+0000 7f4b834647c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:01:59.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:01:59.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:01:59.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:01:59.609 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:01:59.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:00.183 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:02:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:02:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:00.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:00.213 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:00.222+0000 7f0f09baf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:00.246 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:00.254+0000 7f0f09baf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:00.248 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:00.258+0000 7f0f09baf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:00.671 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:02:00.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:00.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:00.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:00.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:00.834+0000 7f0f09baf7c0 -1 Falling back to public interface 2024-04-25T09:02:01.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:02.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:02.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:02.112 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:02:02.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:02.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:02.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:02.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:03.372 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:03.382+0000 7f0f09baf7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:02:03.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:03.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:03.589 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:02:03.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:02:03.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:03.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:04.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:05.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:02:05.260 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:02:05.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:05.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:05.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:06.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:06.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:06.728 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:02:06.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:02:06.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:06.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:07.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:08.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:08.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:08.201 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:02:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:02:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:08.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:08.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:09.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:09.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:09.712 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:02:09.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:02:09.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:09.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:10.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:11.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:11.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:11.273 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:02:11.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:02:11.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:11.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:11.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:12.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:12.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:12.716 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:02:12.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:02:12.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:12.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:13.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:14.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:14.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:14.363 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:02:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:02:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:14.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:15.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:15.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:15.914 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:02:15.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:02:15.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:15.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:16.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:17.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:17.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:17.992 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:02:17.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:02:17.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:17.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:19.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:20.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:20.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:20.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:02:20.226 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:02:20.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:20.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:21.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:22.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:22.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:22.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:02:22.131 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:02:22.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:22.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:23.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:02:24.479 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:02:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:25.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:26.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:26.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:26.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:02:26.359 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:02:26.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:26.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:26.783 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1746944684,v1:127.0.0.1:6819/1746944684] [v2:127.0.0.1:6820/1746944684,v1:127.0.0.1:6821/1746944684] exists,up c17a7836-022c-4706-a0d6-153f9764dcfa 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:02:26.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:26.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:26.788 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:26.788 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:26.788 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:02:26.788 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:02:26.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:02:26.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:02:26.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:02:26.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30354a01-6376-455b-bd96-eabd0a8b1550 2024-04-25T09:02:26.790 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 30354a01-6376-455b-bd96-eabd0a8b1550 2024-04-25T09:02:26.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 30354a01-6376-455b-bd96-eabd0a8b1550' 2024-04-25T09:02:26.791 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:02:26.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiHCpmVpIeMRAANgcsnl9WmZsEEU7mlDOrSQ== 2024-04-25T09:02:26.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiHCpmVpIeMRAANgcsnl9WmZsEEU7mlDOrSQ=="}' 2024-04-25T09:02:26.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30354a01-6376-455b-bd96-eabd0a8b1550 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:02:27.709 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:02:27.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:02:27.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiHCpmVpIeMRAANgcsnl9WmZsEEU7mlDOrSQ== --osd-uuid 30354a01-6376-455b-bd96-eabd0a8b1550 2024-04-25T09:02:27.767 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.778+0000 7f76b91ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:27.770 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.778+0000 7f76b91ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:27.771 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.782+0000 7f76b91ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:27.772 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.782+0000 7f76b91ba7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:02:27.772 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.782+0000 7f76b91ba7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:02:27.772 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.782+0000 7f76b91ba7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:02:27.890 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:27.902+0000 7f76b91ba7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T09:02:33.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T09:02:33.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:02:33.933 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T09:02:33.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:02:33.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:34.507 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T09:02:34.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:02:34.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:34.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:34.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:34.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:34.539 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:34.550+0000 7f8bbbf997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:34.556 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:34.566+0000 7f8bbbf997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:34.557 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:34.566+0000 7f8bbbf997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:34.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:34.973 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:02:34.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:34.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:35.142 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:35.149+0000 7f8bbbf997c0 -1 Falling back to public interface 2024-04-25T09:02:35.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:36.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:36.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:36.450 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:02:36.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:36.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:36.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:36.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:37.644 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:02:37.653+0000 7f8bbbf997c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:02:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:02:37.875 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:02:37.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:37.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:38.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:39.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:39.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:39.565 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:02:39.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:02:39.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:39.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:40.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:41.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:41.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:41.354 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:02:41.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:02:41.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:41.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:41.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:42.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:42.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:42.787 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:02:42.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:02:42.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:42.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:43.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:44.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:44.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:44.243 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:02:44.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:02:44.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:44.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:44.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:45.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:45.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:45.750 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:02:45.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:02:45.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:45.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:46.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:47.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:47.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:47.186 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:02:47.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:02:47.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:47.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:47.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:48.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:48.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:48.722 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:02:48.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:02:48.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:48.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:49.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:50.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:50.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:50.161 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:02:50.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:02:50.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:50.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:51.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:52.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:52.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:52.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:02:52.744 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:02:52.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:52.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:53.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:54.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:54.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:54.576 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:02:54.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:02:54.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:54.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:55.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:56.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:56.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:56.383 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:02:56.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:02:56.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:56.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:56.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:57.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:57.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:57.878 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:02:57.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:02:57.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:57.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:02:58.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:59.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:59.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:59.659 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:02:59.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:02:59.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:59.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:00.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:01.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:01.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:01.833 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:03:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:03:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:01.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:02.512 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1951494756,v1:127.0.0.1:6827/1951494756] [v2:127.0.0.1:6828/1951494756,v1:127.0.0.1:6829/1951494756] exists,up 30354a01-6376-455b-bd96-eabd0a8b1550 2024-04-25T09:03:02.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:03:02.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:03:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:03:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:03:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:03:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:03:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:03:02.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:03:02.515 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:03:02.515 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:03:02.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:03:02.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:03:02.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:03:02.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:03:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:03:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:03:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:03:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:03:02.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:03:03.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:03:03.115 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:03.115 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:03:03.115 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:03:03.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:03:03.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:03.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:03:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803803 2024-04-25T09:03:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803803 2024-04-25T09:03:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803' 2024-04-25T09:03:03.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:03.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:03:03.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574871 2024-04-25T09:03:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574871 2024-04-25T09:03:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803 1-55834574871' 2024-04-25T09:03:03.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:03.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:03:03.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-04-25T09:03:03.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-04-25T09:03:03.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803 1-55834574871 2-85899345933' 2024-04-25T09:03:03.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:03.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:03:04.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953481 2024-04-25T09:03:04.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953481 2024-04-25T09:03:04.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803803 1-55834574871 2-85899345933 3-137438953481' 2024-04-25T09:03:04.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:04.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:04.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803803 2024-04-25T09:03:04.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:03:04.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803803 2024-04-25T09:03:04.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:04.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803803 2024-04-25T09:03:04.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-04-25T09:03:04.069 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803803 2024-04-25T09:03:04.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:04.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803803 -lt 25769803803 2024-04-25T09:03:04.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:04.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574871 2024-04-25T09:03:04.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:04.511 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:03:04.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:04.513 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574871 2024-04-25T09:03:04.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574871 2024-04-25T09:03:04.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-04-25T09:03:04.514 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574871 2024-04-25T09:03:04.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:05.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574871 -lt 55834574871 2024-04-25T09:03:05.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:05.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345933 2024-04-25T09:03:05.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:05.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:03:05.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345933 2024-04-25T09:03:05.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:05.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-04-25T09:03:05.014 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345933 2024-04-25T09:03:05.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345933' 2024-04-25T09:03:05.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:03:05.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-04-25T09:03:05.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:05.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953481 2024-04-25T09:03:05.486 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:05.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:03:05.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953481 2024-04-25T09:03:05.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:05.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953481 2024-04-25T09:03:05.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953481' 2024-04-25T09:03:05.490 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953481 2024-04-25T09:03:05.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:03:05.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953479 -lt 137438953481 2024-04-25T09:03:05.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:03:06.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:03:06.971 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:03:07.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953481 -lt 137438953481 2024-04-25T09:03:07.394 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:03:07.395 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:07.395 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:07.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:03:07.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:03:07.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:03:07.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:03:07.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:03:07.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:03:07.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:03:07.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:03:08.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:03:08.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:03:08.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:08.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:08.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:03:08.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:03:08.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:03:08.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T09:03:08.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-04-25T09:03:08.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T09:03:09.523 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'pool-rep' already exists 2024-04-25T09:03:09.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:03:10.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:03:10.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:03:10.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:03:10.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:03:10.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:03:10.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:03:10.673 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:11.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:03:11.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803806 2024-04-25T09:03:11.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803806 2024-04-25T09:03:11.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803806' 2024-04-25T09:03:11.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:11.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:03:11.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574874 2024-04-25T09:03:11.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574874 2024-04-25T09:03:11.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803806 1-55834574874' 2024-04-25T09:03:11.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:11.455 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:03:11.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345936 2024-04-25T09:03:11.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345936 2024-04-25T09:03:11.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803806 1-55834574874 2-85899345936' 2024-04-25T09:03:11.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:11.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:03:11.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953484 2024-04-25T09:03:11.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953484 2024-04-25T09:03:11.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803806 1-55834574874 2-85899345936 3-137438953484' 2024-04-25T09:03:11.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:11.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803806 2024-04-25T09:03:11.787 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:11.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:03:11.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803806 2024-04-25T09:03:11.789 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:11.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803806 2024-04-25T09:03:11.790 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803806 2024-04-25T09:03:11.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-04-25T09:03:11.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803806 -lt 25769803806 2024-04-25T09:03:12.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:12.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574874 2024-04-25T09:03:12.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:12.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:03:12.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574874 2024-04-25T09:03:12.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:12.226 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574874 2024-04-25T09:03:12.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574874 2024-04-25T09:03:12.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-04-25T09:03:12.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:12.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574874 -lt 55834574874 2024-04-25T09:03:12.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:12.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345936 2024-04-25T09:03:12.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:12.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:03:12.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345936 2024-04-25T09:03:12.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:12.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345936 2024-04-25T09:03:12.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345936' 2024-04-25T09:03:12.682 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345936 2024-04-25T09:03:12.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:03:13.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345937 -lt 85899345936 2024-04-25T09:03:13.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:13.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:13.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953484 2024-04-25T09:03:13.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:03:13.125 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953484 2024-04-25T09:03:13.125 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:13.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953484 2024-04-25T09:03:13.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953484' 2024-04-25T09:03:13.127 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953484 2024-04-25T09:03:13.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:03:13.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953484 -lt 137438953484 2024-04-25T09:03:13.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:03:13.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:13.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:14.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:03:14.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:03:14.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:03:14.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:03:14.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:03:14.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:03:14.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:03:14.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:03:14.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:03:14.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:03:14.608 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:14.609 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-04-25T09:03:15.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-04-25T09:03:15.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:15.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:15.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:15.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:15.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:15.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:15.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:15.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:15.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:15.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:15.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:15.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:15.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:15.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:15.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:15.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:03:15.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:15.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:03:15.349 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:15.349 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:03:15.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:15.824 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:15.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:15.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:15.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:15.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:15.826 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:15.827 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:15.827 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:15.827 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:15.827 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:15.828 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:15.828 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:15.828 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:15.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:15.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:15.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:15.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:15.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:16.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:16.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:16.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:16.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:16.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:03:16.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:16.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:16.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:16.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:16.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:03:16.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:16.325 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:03:16.325 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:03:16.325 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:16.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:16.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:16.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:16.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:16.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:16.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:16.769 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:16.770 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:16.770 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:16.770 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:16.770 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:16.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:16.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:16.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:16.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:16.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:16.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:16.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:16.892 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:16.892 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:16.893 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:16.893 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:16.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:16.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:17.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:17.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:17.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:17.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:17.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:17.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:17.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:17.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:03:17.195 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:17.195 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:03:17.612 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:17.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:17.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:17.617 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:17.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:17.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:17.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:17.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:17.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:17.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:17.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:17.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:17.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:17.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:17.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:17.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:17.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:17.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:17.894 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:17.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:17.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:03:17.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:17.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:17.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:17.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:18.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:18.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:18.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:03:18.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:18.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:03:18.072 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:03:18.072 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:18.496 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:18.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:18.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:18.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:18.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:18.501 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:18.501 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:18.501 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:18.501 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:18.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:18.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:18.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:18.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:18.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:18.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:18.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:18.629 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:18.629 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:18.629 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:18.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:18.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:18.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:03:18.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:18.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:18.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:18.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:18.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:18.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:19.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:19.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:19.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:19.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:03:19.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:19.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:03:19.027 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:03:19.027 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:19.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:19.501 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:19.501 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:19.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:19.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:19.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:19.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:19.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:19.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:19.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:19.504 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:19.504 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:19.504 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:19.505 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:19.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:19.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:19.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:19.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:19.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:19.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:19.635 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:19.636 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:19.636 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:19.636 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:19.636 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:19.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:19.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:03:19.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:19.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:19.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:19.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:19.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:19.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:19.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:19.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:19.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:03:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:03:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:20.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:20.404 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:20.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:20.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:20.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:20.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:20.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:20.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:20.406 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:20.407 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:20.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:20.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:20.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:20.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:20.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:20.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:03:20.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:03:20.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:20.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:20.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T09:03:20.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:20.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:20.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:20.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:20.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:20.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:20.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:20.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:03:20.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:20.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:20.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:20.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:20.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:20.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:03:20.857 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:03:20.857 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:21.356 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:21.358 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:21.359 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:21.359 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:21.359 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:21.360 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:21.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:21.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:21.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:21.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:21.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:21.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:21.478 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:21.479 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:21.479 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:21.479 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:21.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:21.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:21.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:03:21.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:03:21.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:21.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:21.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T09:03:21.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:21.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:21.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:21.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:21.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:21.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:21.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:21.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:21.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:21.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:03:21.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:21.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:21.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:21.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:21.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:21.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:03:21.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:21.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:03:21.826 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:03:21.827 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:22.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:22.337 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:22.337 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:22.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:22.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:22.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:22.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:22.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:22.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:22.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:22.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:22.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:22.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:22.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:22.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:22.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:22.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:22.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:22.468 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:22.469 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:22.469 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:22.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:22.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:22.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:03:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:03:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T09:03:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:22.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:22.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:22.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:22.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:22.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:22.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:22.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:03:22.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:22.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:22.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:22.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:22.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:22.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:22.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:22.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:22.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:22.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:03:22.806 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:22.806 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:03:23.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:23.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:23.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:23.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:23.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:23.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:23.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:23.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:23.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:23.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:23.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:23.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:23.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:23.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:23.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:23.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:23.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:03:23.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:03:23.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:23.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:23.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T09:03:23.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:23.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:23.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:23.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:23.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:23.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:23.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:23.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:23.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:23.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:23.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:23.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:23.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:23.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:03:23.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:23.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:03:23.683 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:03:23.684 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:24.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:24.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:24.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:24.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:24.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:24.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:24.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:24.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:24.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:24.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:24.163 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:24.163 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:24.163 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:24.163 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:24.163 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:24.164 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:24.164 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:24.164 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:24.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:24.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:24.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:24.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:24.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:24.283 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:24.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:03:24.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:03:24.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:24.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:24.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T09:03:24.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:24.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:24.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:24.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:24.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:24.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:24.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:24.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:03:24.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:24.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:03:24.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-04-25T09:03:24.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:03:24.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:24.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:03:24.603 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:25.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:25.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:25.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:25.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:25.100 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:25.100 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:25.100 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:25.100 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:25.100 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:25.100 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:25.101 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:25.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:25.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:25.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:25.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:25.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:25.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:25.224 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:25.225 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:25.225 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:25.225 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:25.225 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:25.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:25.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:25.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:25.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:25.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:25.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-04-25T09:03:25.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T09:03:25.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-04-25T09:03:25.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:03:25.387 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:03:25.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-04-25T09:03:25.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-04-25T09:03:25.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:25.821 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:03:25.822 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:03:25.822 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:03:26.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:03:26.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:03:26.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:03:26.246 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:03:26.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:03:26.246 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:03:26.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:03:26.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:03:26.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:03:26.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:03:26.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:03:26.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:03:26.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:03:26.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:03:26.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:03:26.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:03:26.807 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:26.807 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:03:26.807 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:03:26.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:03:26.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:26.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:03:26.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803811 2024-04-25T09:03:26.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803811 2024-04-25T09:03:26.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803811' 2024-04-25T09:03:26.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:26.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:03:27.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574879 2024-04-25T09:03:27.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574879 2024-04-25T09:03:27.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803811 1-55834574879' 2024-04-25T09:03:27.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:27.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:03:27.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T09:03:27.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T09:03:27.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803811 1-55834574879 2-85899345942' 2024-04-25T09:03:27.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:27.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:03:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953490 2024-04-25T09:03:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953490 2024-04-25T09:03:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803811 1-55834574879 2-85899345942 3-137438953490' 2024-04-25T09:03:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:27.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803811 2024-04-25T09:03:27.439 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:27.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:03:27.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803811 2024-04-25T09:03:27.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:27.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803811 2024-04-25T09:03:27.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-04-25T09:03:27.442 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803811 2024-04-25T09:03:27.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:27.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803809 -lt 25769803811 2024-04-25T09:03:27.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:03:28.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:03:28.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:29.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803812 -lt 25769803811 2024-04-25T09:03:29.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:29.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574879 2024-04-25T09:03:29.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:29.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:03:29.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574879 2024-04-25T09:03:29.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:29.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574879 2024-04-25T09:03:29.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-04-25T09:03:29.338 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574879 2024-04-25T09:03:29.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:29.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574880 -lt 55834574879 2024-04-25T09:03:29.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:29.755 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345942 2024-04-25T09:03:29.755 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:29.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:03:29.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345942 2024-04-25T09:03:29.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:29.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T09:03:29.759 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345942 2024-04-25T09:03:29.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-04-25T09:03:29.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:03:30.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345942 2024-04-25T09:03:30.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:30.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953490 2024-04-25T09:03:30.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:30.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:03:30.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953490 2024-04-25T09:03:30.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:30.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953490 2024-04-25T09:03:30.186 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953490 2024-04-25T09:03:30.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953490' 2024-04-25T09:03:30.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:03:30.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953490 -lt 137438953490 2024-04-25T09:03:30.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:03:30.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:30.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:31.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:03:31.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:03:31.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:03:31.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:03:31.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:03:31.235 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:03:31.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:03:31.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:03:31.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:03:31.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:03:31.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:31.651 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:32.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:03:32.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:03:32.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:03:32.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T09:03:32.197 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:03:32.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-04-25T09:03:32.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-04-25T09:03:32.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T09:03:32.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:03:32.361 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:32.785 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:03:32.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803815 2024-04-25T09:03:32.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803815 2024-04-25T09:03:32.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803815' 2024-04-25T09:03:32.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:32.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:03:33.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574883 2024-04-25T09:03:33.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574883 2024-04-25T09:03:33.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803815 1-55834574883' 2024-04-25T09:03:33.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:33.105 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:03:33.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345945 2024-04-25T09:03:33.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345945 2024-04-25T09:03:33.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803815 1-55834574883 2-85899345945' 2024-04-25T09:03:33.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:33.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:03:33.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953493 2024-04-25T09:03:33.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953493 2024-04-25T09:03:33.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803815 1-55834574883 2-85899345945 3-137438953493' 2024-04-25T09:03:33.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:33.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803815 2024-04-25T09:03:33.409 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:33.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:03:33.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803815 2024-04-25T09:03:33.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:33.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803815 2024-04-25T09:03:33.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-04-25T09:03:33.412 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803815 2024-04-25T09:03:33.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:33.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803812 -lt 25769803815 2024-04-25T09:03:33.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:03:34.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:03:34.822 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:35.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803815 -lt 25769803815 2024-04-25T09:03:35.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:35.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574883 2024-04-25T09:03:35.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:35.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:03:35.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574883 2024-04-25T09:03:35.300 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:35.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574883 2024-04-25T09:03:35.301 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574883 2024-04-25T09:03:35.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574883' 2024-04-25T09:03:35.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:35.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574883 -lt 55834574883 2024-04-25T09:03:35.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:35.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345945 2024-04-25T09:03:35.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:35.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:03:35.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:35.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345945 2024-04-25T09:03:35.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345945 2024-04-25T09:03:35.777 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345945 2024-04-25T09:03:35.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345945' 2024-04-25T09:03:35.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:03:36.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345945 -lt 85899345945 2024-04-25T09:03:36.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:36.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:36.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953493 2024-04-25T09:03:36.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:03:36.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953493 2024-04-25T09:03:36.223 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:36.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953493 2024-04-25T09:03:36.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953493' 2024-04-25T09:03:36.225 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953493 2024-04-25T09:03:36.226 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:03:36.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953493 -lt 137438953493 2024-04-25T09:03:36.657 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T09:03:36.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:03:37.062 INFO:tasks.workunit.client.0.smithi031.stderr:dumped all 2024-04-25T09:03:37.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-04-25T09:03:37.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-04-25T09:03:37.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-04-25T09:03:37.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T09:03:37.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:03:37.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-04-25T09:03:38.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-04-25T09:03:38.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-04-25T09:03:38.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-04-25T09:03:38.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T09:03:39.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-04-25T09:03:39.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-04-25T09:03:39.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:39.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:39.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:39.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:39.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:39.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:39.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:03:39.485 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:03:39.485 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:40.010 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:40.010 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:40.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:40.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:40.015 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:40.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:40.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:40.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:40.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:40.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:40.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:40.136 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:40.136 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:40.137 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:40.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:40.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:40.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:03:40.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:40.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:40.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:40.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:40.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:03:40.256 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:40.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:40.726 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:40.726 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:40.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:40.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:40.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:40.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:40.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:40.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:40.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:40.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:40.729 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:40.729 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:40.729 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:40.729 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:40.730 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:40.730 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:40.730 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:40.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:40.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:40.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:40.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:40.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:40.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:40.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:40.854 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:40.854 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:40.854 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:40.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:40.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:40.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:03:40.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:40.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:40.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:40.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:40.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:40.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:40.975 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:40.975 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:40.975 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:40.975 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:40.976 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:40.976 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:40.976 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:40.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:41.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:41.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:03:41.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:41.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:41.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T09:03:41.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:41.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:41.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:41.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:41.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:41.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:41.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:41.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:03:41.201 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:03:41.201 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:41.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:41.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:41.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:41.682 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:41.683 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:41.683 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:41.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:41.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:41.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:41.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:41.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:41.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:41.805 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:41.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:03:41.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:41.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:41.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:41.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:41.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:03:41.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:41.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:03:41.932 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:03:41.932 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:42.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:42.374 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:42.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:42.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:42.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:42.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:42.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:42.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:42.376 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:42.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:42.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:42.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:42.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:42.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:42.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:42.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:42.498 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:42.499 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:42.499 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:42.499 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:42.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:42.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:03:42.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:42.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:42.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:42.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:42.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:42.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:42.629 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:42.630 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:42.631 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:42.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:42.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:42.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:03:42.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:42.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:42.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T09:03:42.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:42.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:42.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:42.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:42.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:42.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:42.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:42.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:42.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:03:42.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:42.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:03:42.826 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:03:42.826 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:43.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:43.250 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:43.250 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:43.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:43.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:43.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:43.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:43.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:43.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:43.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:43.253 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:43.254 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:43.254 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:43.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:43.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:43.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:43.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:43.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:43.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:43.492 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:03:43.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:43.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:43.986 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:43.986 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:43.986 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:43.986 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:43.987 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:44.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:44.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:44.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:44.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:44.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:44.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:44.115 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:44.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:44.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:44.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:03:44.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:44.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:44.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:44.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:44.234 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:44.234 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:44.235 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:44.235 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:44.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:44.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:44.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:44.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:03:44.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:03:44.438 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:44.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:44.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:44.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:44.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:44.862 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:44.862 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:44.862 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:44.863 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:44.863 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:44.863 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:44.863 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:44.863 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:44.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:44.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:44.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:44.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:44.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:44.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:44.992 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:44.992 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:44.992 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:44.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:44.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:44.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:03:45.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:45.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:45.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:03:45.116 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:03:45.116 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:45.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:45.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:45.562 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:45.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:45.564 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:45.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:45.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:45.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:45.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:45.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:45.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:45.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:45.678 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:45.678 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:45.679 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:45.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:45.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:45.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:45.801 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:45.801 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:45.801 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:45.801 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:45.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:03:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T09:03:45.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:45.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:45.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:45.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:45.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:45.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:45.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:03:45.968 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:46.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:46.415 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:46.415 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:46.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:46.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:46.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:46.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:46.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:46.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:46.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:46.417 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:46.418 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:46.418 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:46.419 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:46.419 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:46.419 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:46.419 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:46.419 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:46.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:46.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:46.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:46.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:46.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:46.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:46.541 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:46.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:46.542 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:46.543 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:46.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:46.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:46.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:03:46.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:46.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:46.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:46.665 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:03:46.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:03:46.666 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:47.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:47.169 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:47.169 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:47.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:47.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:47.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:47.172 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:47.173 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:47.173 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:47.173 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:47.173 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:47.173 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:47.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:47.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:47.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:47.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:47.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:47.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:47.292 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:47.292 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:47.292 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:47.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:47.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:47.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:03:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:47.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:47.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:47.419 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:47.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:47.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:47.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:47.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:47.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:47.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:03:47.613 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:48.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:48.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:48.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:48.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:48.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:48.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:48.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:48.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:48.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:48.118 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:48.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:48.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:48.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:48.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:48.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:48.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:48.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:48.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:48.250 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:48.250 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:48.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:48.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:03:48.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:48.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:48.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:48.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:03:48.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:03:48.373 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:48.838 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:48.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:48.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:48.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:48.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:48.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:48.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:48.840 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:48.840 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:48.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:48.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:48.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:48.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:48.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:48.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:48.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:48.957 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:48.957 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:48.957 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:48.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:48.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:48.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:03:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:49.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:49.086 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:49.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:49.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:49.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:03:49.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:49.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:49.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T09:03:49.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:49.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:49.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:49.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:49.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:49.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:49.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:49.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:49.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:03:49.274 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:49.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:03:49.275 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:03:49.275 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:49.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:49.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:49.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:49.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:49.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:49.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:49.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:49.692 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:49.693 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:49.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:49.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:49.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:49.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:49.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:49.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:49.809 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:49.810 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:49.810 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:49.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:49.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:49.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:03:49.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:49.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:49.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:49.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:49.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:49.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:49.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:49.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:03:49.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:49.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:03:49.927 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:03:49.927 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:50.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:50.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:50.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:50.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:50.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:50.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:50.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:50.368 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:50.368 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:50.368 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:50.368 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:50.369 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:50.369 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.369 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:50.369 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:50.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:50.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:50.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:50.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:50.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:50.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:50.482 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:50.482 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:50.482 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.482 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:50.482 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:50.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:50.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:03:50.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:50.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:50.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:50.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:50.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:50.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:50.598 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:50.598 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:50.598 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:50.598 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:50.599 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.599 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:50.599 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:50.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:50.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:50.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:03:50.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:50.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:50.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T09:03:50.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:50.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:50.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:50.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:50.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:50.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:50.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:50.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:50.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:50.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:50.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:03:50.759 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:51.210 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:51.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:51.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:51.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:51.214 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:51.214 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:51.214 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:51.214 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:51.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:51.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:51.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:51.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:51.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:51.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:03:51.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:51.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:51.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:51.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:51.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:03:51.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:51.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:03:51.453 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:51.453 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:03:51.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:51.930 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:51.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:51.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:51.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:51.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:51.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:51.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:51.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:51.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:52.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:52.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:52.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:52.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:52.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:52.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:52.061 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:52.061 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:52.061 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:52.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:52.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:52.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:03:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:52.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:52.188 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:52.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:52.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:52.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:52.189 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:52.190 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:52.190 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:52.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:52.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:52.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:52.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:52.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:52.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:52.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:52.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:03:52.360 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:52.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:52.826 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:52.827 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:52.827 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:52.827 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:52.827 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:52.827 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:52.828 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:52.828 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:52.828 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:52.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:52.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:52.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:52.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:52.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:52.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:03:53.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:03:53.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:53.074 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:03:53.074 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:03:53.074 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:53.497 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:53.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:53.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:53.500 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:53.501 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:53.501 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:53.501 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:53.501 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:53.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:53.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:53.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:53.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:53.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:53.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:53.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:53.618 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:53.619 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:53.619 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:53.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:53.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:53.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:03:53.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:53.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:53.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:53.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:53.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:53.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:53.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:53.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:53.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:53.739 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:53.739 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:53.739 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:53.739 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:53.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:53.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:53.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:53.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:53.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:53.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:53.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:53.908 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:03:53.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:53.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:03:53.909 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:03:53.909 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:54.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:54.388 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:54.388 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:54.388 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:54.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:54.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:54.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:54.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:54.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:54.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:54.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:54.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:54.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:54.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:54.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:54.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:54.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:54.391 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:54.391 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:54.391 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:54.392 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:54.392 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:54.392 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:54.393 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:54.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:54.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:54.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:54.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:54.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:54.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:54.514 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:54.515 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:54.515 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:54.515 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:54.515 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:54.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:54.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:54.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:03:54.642 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:55.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:55.123 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:55.123 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:55.123 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:55.124 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:55.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:55.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:55.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:55.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:55.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:55.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:55.126 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:55.126 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:55.126 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:55.126 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:55.126 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:55.126 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:55.127 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:55.127 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:55.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:55.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:55.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:55.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:55.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:55.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:55.258 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:55.258 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:55.258 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:55.259 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:55.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:55.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:03:55.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:55.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:55.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:55.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:55.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:55.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:55.380 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:55.380 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:55.380 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:55.381 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:55.381 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:55.381 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:55.382 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:55.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:55.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:55.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:55.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:55.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:55.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:03:55.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:55.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:03:55.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:55.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:03:55.551 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:03:55.551 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:55.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:55.970 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:55.970 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:55.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:03:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:03:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:55.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:55.973 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:55.973 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:55.973 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:55.973 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:55.974 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:55.974 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:55.974 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:55.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:56.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:56.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:56.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:56.092 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:03:56.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:56.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:56.093 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:56.093 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:56.094 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:56.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:56.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:56.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:03:56.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-04-25T09:03:56.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:03:56.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:03:56.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:03:56.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:03:56.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:03:56.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:03:56.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:03:56.222 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:03:56.222 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:03:56.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:03:56.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:03:56.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:03:56.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:03:56.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:56.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:03:56.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:03:56.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:03:56.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:56.688 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:03:56.688 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:56.688 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:56.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:03:56.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:56.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:03:56.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:03:56.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:03:56.816 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:03:56.817 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:03:56.817 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:56.817 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:56.817 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:56.818 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:03:56.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:03:56.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:03:56.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:03:56.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:03:56.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:03:56.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:03:56.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:03:56.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:03:56.933 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:03:56.933 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:03:56.933 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:03:56.933 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:56.934 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:56.934 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:03:56.934 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.3.asok 2024-04-25T09:03:56.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:03:57.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:57.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:57.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:03:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:03:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:03:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:03:57.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:03:57.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:03:57.091 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:03:57.091 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:03:57.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:03:57.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:03:57.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:03:57.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:03:57.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:03:57.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:03:57.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:03:57.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:03:57.174 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:03:57.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:03:57.581 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:03:57.581 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:03:57.581 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:03:57.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:03:57.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:57.582 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:03:57.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803822 2024-04-25T09:03:57.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803822 2024-04-25T09:03:57.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803822' 2024-04-25T09:03:57.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:57.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:03:57.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574890 2024-04-25T09:03:57.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574890 2024-04-25T09:03:57.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803822 1-55834574890' 2024-04-25T09:03:57.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:57.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:03:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345952 2024-04-25T09:03:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345952 2024-04-25T09:03:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803822 1-55834574890 2-85899345952' 2024-04-25T09:03:58.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:58.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:03:58.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953500 2024-04-25T09:03:58.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953500 2024-04-25T09:03:58.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803822 1-55834574890 2-85899345952 3-137438953500' 2024-04-25T09:03:58.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:58.271 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803822 2024-04-25T09:03:58.271 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:58.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:03:58.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803822 2024-04-25T09:03:58.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:58.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803822 2024-04-25T09:03:58.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-04-25T09:03:58.274 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803822 2024-04-25T09:03:58.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:58.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803822 -lt 25769803822 2024-04-25T09:03:58.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:58.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574890 2024-04-25T09:03:58.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:58.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:03:58.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574890 2024-04-25T09:03:58.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:58.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574890 2024-04-25T09:03:58.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574890' 2024-04-25T09:03:58.687 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574890 2024-04-25T09:03:58.687 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:59.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574889 -lt 55834574890 2024-04-25T09:03:59.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:00.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:00.109 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:00.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574890 -lt 55834574890 2024-04-25T09:04:00.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:00.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:00.591 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345952 2024-04-25T09:04:00.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:00.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345952 2024-04-25T09:04:00.593 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:00.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345952 2024-04-25T09:04:00.595 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345952 2024-04-25T09:04:00.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345952' 2024-04-25T09:04:00.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:01.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345952 -lt 85899345952 2024-04-25T09:04:01.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:01.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953500 2024-04-25T09:04:01.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:01.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:01.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953500 2024-04-25T09:04:01.014 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:01.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953500 2024-04-25T09:04:01.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953500' 2024-04-25T09:04:01.015 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953500 2024-04-25T09:04:01.016 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:01.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953500 -lt 137438953500 2024-04-25T09:04:01.458 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:04:01.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:01.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:02.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:04:02.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:04:02.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:04:02.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:04:02.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:04:02.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:04:02.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:04:02.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:04:02.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:04:02.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:04:02.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:02.454 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:03.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:04:03.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:04:03.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:04:03.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T09:04:03.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:04:03.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-04-25T09:04:03.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T09:04:03.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-04-25T09:04:03.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T09:04:03.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:03.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:03.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:03.619 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:04:03.619 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:04:03.619 INFO:tasks.workunit.client.0.smithi031.stderr:3' 2024-04-25T09:04:03.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:03.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:03.620 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:03.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803825 2024-04-25T09:04:03.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803825 2024-04-25T09:04:03.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803825' 2024-04-25T09:04:03.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:03.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:03.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574893 2024-04-25T09:04:03.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574893 2024-04-25T09:04:03.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803825 1-55834574893' 2024-04-25T09:04:03.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:03.929 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:04.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345955 2024-04-25T09:04:04.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345955 2024-04-25T09:04:04.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803825 1-55834574893 2-85899345955' 2024-04-25T09:04:04.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:04.080 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:04.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953503 2024-04-25T09:04:04.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953503 2024-04-25T09:04:04.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803825 1-55834574893 2-85899345955 3-137438953503' 2024-04-25T09:04:04.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:04.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803825 2024-04-25T09:04:04.262 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:04.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:04.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803825 2024-04-25T09:04:04.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:04.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803825 2024-04-25T09:04:04.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-04-25T09:04:04.265 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803825 2024-04-25T09:04:04.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:04.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803825 -lt 25769803825 2024-04-25T09:04:04.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:04.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574893 2024-04-25T09:04:04.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:04.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:04.731 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574893 2024-04-25T09:04:04.731 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:04.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574893 2024-04-25T09:04:04.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574893' 2024-04-25T09:04:04.733 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574893 2024-04-25T09:04:04.733 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:05.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574893 -lt 55834574893 2024-04-25T09:04:05.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:05.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345955 2024-04-25T09:04:05.162 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:05.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:05.164 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345955 2024-04-25T09:04:05.165 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:05.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345955 2024-04-25T09:04:05.166 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 85899345955 2024-04-25T09:04:05.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345955' 2024-04-25T09:04:05.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:05.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345953 -lt 85899345955 2024-04-25T09:04:05.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:06.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:06.602 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:07.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345955 -lt 85899345955 2024-04-25T09:04:07.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:07.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953503 2024-04-25T09:04:07.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:07.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:07.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953503 2024-04-25T09:04:07.043 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:07.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953503 2024-04-25T09:04:07.044 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 137438953503 2024-04-25T09:04:07.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953503' 2024-04-25T09:04:07.044 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:07.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953503 -lt 137438953503 2024-04-25T09:04:07.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:04:07.508 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T09:04:07.926 INFO:tasks.workunit.client.0.smithi031.stderr:dumped all 2024-04-25T09:04:07.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-04-25T09:04:07.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-04-25T09:04:07.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-04-25T09:04:07.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-04-25T09:04:07.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-04-25T09:04:07.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-04-25T09:04:07.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-04-25T09:04:08.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-04-25T09:04:08.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-04-25T09:04:09.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-04-25T09:04:09.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:04:09.468 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T09:04:09.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-04-25T09:04:10.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:04:10.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-04-25T09:04:10.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-04-25T09:04:10.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T09:04:10.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T09:04:11.118 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'pool-rep' does not exist 2024-04-25T09:04:11.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:04:11.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:04:11.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:04:11.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:04:11.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:04:11.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:04:11.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:04:11.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:04:11.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:04:11.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:04:11.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:04:11.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:04:11.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:04:11.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:04:11.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:04:11.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:04:11.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:04:11.452 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:04:11.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:04:11.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:04:11.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:04:11.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:11.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:11.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:04:11.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:04:11.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:04:11.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:04:11.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:04:11.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:04:11.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:04:11.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:04:11.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:04:11.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:04:11.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:04:11.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:04:11.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:04:11.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:04:11.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:04:11.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:04:11.494 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:04:11.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:04:11.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:04:11.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:04:11.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:11.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:11.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:04:11.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:04:11.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:04:11.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:04:11.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:11.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:11.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383712 2024-04-25T09:04:11.502 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:04:11.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:04:11.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:04:11.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:04:11.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:04:11.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:04:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:04:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:04:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:04:11.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:04:11.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:04:11.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:04:11.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:04:11.979 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:04:11.979 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:11.979 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:11.979 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:04:11.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:04:12.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:04:12.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:04:12.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:04:12.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:04:12.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:04:12.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:04:12.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:04:12.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:04:12.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:04:12.191 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:12.191 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:12.191 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:12.191 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T09:04:12.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:04:12.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get fsid 2024-04-25T09:04:12.305 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:04:12.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:04:12.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:04:12.306 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:12.307 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T09:04:12.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:04:12.308 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get mon_host 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:04:12.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:04:13.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:04:13.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:04:13.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:04:13.340 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:04:13.340 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:13.340 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:13.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:04:13.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:04:13.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:04:13.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:04:13.692 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T09:04:13.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:04:13.712 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T09:04:13.713 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T09:04:13.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:04:13.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:04:13.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:13.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:04:13.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:04:13.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:04:13.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:04:13.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:04:13.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:04:13.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:04:13.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:04:13.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:04:13.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:04:13.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:04:13.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:13.723 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:13.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:04:13.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:04:13.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:04:13.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:04:13.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:04:13.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:04:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:04:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:04:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:04:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:04:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:04:13.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:04:13.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:04:13.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:04:13.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:04:13.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e6577b9e-d226-4e7a-b953-3f6b96e34685 2024-04-25T09:04:13.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e6577b9e-d226-4e7a-b953-3f6b96e34685' 2024-04-25T09:04:13.727 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 e6577b9e-d226-4e7a-b953-3f6b96e34685 2024-04-25T09:04:13.727 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:04:13.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNHCpm8h4NLRAA/+5iXPat2/HKq1PmF1yphA== 2024-04-25T09:04:13.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNHCpm8h4NLRAA/+5iXPat2/HKq1PmF1yphA=="}' 2024-04-25T09:04:13.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e6577b9e-d226-4e7a-b953-3f6b96e34685 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:04:14.044 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:04:14.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:04:14.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNHCpm8h4NLRAA/+5iXPat2/HKq1PmF1yphA== --osd-uuid e6577b9e-d226-4e7a-b953-3f6b96e34685 2024-04-25T09:04:14.093 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.100+0000 7fc577c567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:14.112 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.120+0000 7fc577c567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:14.114 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.120+0000 7fc577c567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:14.114 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.120+0000 7fc577c567c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:04:14.114 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.120+0000 7fc577c567c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:04:14.114 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.120+0000 7fc577c567c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:04:14.152 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:14.160+0000 7fc577c567c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:04:19.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:04:19.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:04:19.906 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:04:19.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:04:19.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:04:20.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:04:20.481 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:04:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:04:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:04:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:04:20.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:04:20.516 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:20.528+0000 7f0640a707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:20.550 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:20.560+0000 7f0640a707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:20.552 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:20.564+0000 7f0640a707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:20.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:04:20.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:04:20.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:04:20.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:04:20.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:04:20.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:20.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:04:20.947 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:04:20.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:20.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:21.120 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:21.132+0000 7f0640a707c0 -1 Falling back to public interface 2024-04-25T09:04:21.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:22.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:22.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:22.367 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:04:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:04:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:22.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:22.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:23.690 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:23.700+0000 7f0640a707c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:04:23.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:23.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:23.859 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:04:23.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:04:23.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:23.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:24.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:25.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:25.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:25.418 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:04:25.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:04:25.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:25.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:25.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:26.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:26.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:26.864 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:04:26.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:04:26.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:26.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:27.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:28.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:28.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:28.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:04:28.405 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:04:28.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:28.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:28.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:29.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:29.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:29.862 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:04:29.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:04:29.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:29.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:30.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:31.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:31.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:31.432 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:04:31.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:04:31.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:31.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:31.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:32.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:32.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:32.886 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:04:32.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:04:32.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:32.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:33.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:34.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:34.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:34.395 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:04:34.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:04:34.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:34.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:34.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:35.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:35.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:35.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:04:35.914 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:04:35.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:35.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:36.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:37.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:37.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:37.443 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:04:37.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:04:37.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:37.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:38.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:39.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:39.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:39.456 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:04:39.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:04:39.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:39.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:40.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:41.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:41.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:41.175 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:04:41.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:04:41.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:41.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:41.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:42.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:04:42.983 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:04:42.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:42.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:44.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:45.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:45.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:45.032 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:04:45.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:04:45.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:45.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:45.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:46.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:46.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:46.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:04:46.595 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:04:46.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:46.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:47.047 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3913924380,v1:127.0.0.1:6803/3913924380] [v2:127.0.0.1:6804/3913924380,v1:127.0.0.1:6805/3913924380] exists,up e6577b9e-d226-4e7a-b953-3f6b96e34685 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:47.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:04:47.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:04:47.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:04:47.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f9139018-e36b-4b94-90fd-d11736267f71 2024-04-25T09:04:47.054 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 f9139018-e36b-4b94-90fd-d11736267f71 2024-04-25T09:04:47.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f9139018-e36b-4b94-90fd-d11736267f71' 2024-04-25T09:04:47.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:04:47.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvHCpm9r0EBRAAOxz7T2bEh3B+kKAW1P7rwA== 2024-04-25T09:04:47.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvHCpm9r0EBRAAOxz7T2bEh3B+kKAW1P7rwA=="}' 2024-04-25T09:04:47.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f9139018-e36b-4b94-90fd-d11736267f71 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:04:47.551 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:04:47.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:04:47.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvHCpm9r0EBRAAOxz7T2bEh3B+kKAW1P7rwA== --osd-uuid f9139018-e36b-4b94-90fd-d11736267f71 2024-04-25T09:04:47.607 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.619+0000 7f08c69427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:47.609 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.619+0000 7f08c69427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:47.611 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.623+0000 7f08c69427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:47.611 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.623+0000 7f08c69427c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:04:47.611 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.623+0000 7f08c69427c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:04:47.611 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.623+0000 7f08c69427c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:04:47.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:47.667+0000 7f08c69427c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:04:53.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:04:53.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:04:53.758 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:04:53.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:04:53.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:04:54.347 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:04:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:04:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:04:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:04:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:04:54.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:04:54.382 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:54.395+0000 7f73cbddf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:54.385 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:54.395+0000 7f73cbddf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:54.386 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:54.395+0000 7f73cbddf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:54.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:04:54.818 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:04:54.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:54.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:54.952 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:54.963+0000 7f73cbddf7c0 -1 Falling back to public interface 2024-04-25T09:04:55.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:56.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:56.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:56.272 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:04:56.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:04:56.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:56.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:56.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:57.454 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:04:57.467+0000 7f73cbddf7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:04:57.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:57.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:57.728 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:04:57.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:04:57.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:57.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:58.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:59.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:59.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:59.243 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:04:59.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:04:59.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:59.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:59.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:00.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:00.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:00.716 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:05:00.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:05:00.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:00.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:01.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:02.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:02.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:02.265 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:05:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:05:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:02.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:03.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:03.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:03.798 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:05:03.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:05:03.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:03.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:04.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:05.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:05.364 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:05:05.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:05:05.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:05.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:05.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:06.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:06.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:06.820 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:05:06.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:05:06.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:06.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:07.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:08.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:08.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:08.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:05:08.383 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:05:08.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:08.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:08.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:09.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:09.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:09.891 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:05:09.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:05:09.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:09.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:11.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:12.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:12.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:12.047 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:05:12.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:05:12.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:12.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:12.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:13.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:13.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:13.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:05:13.804 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:05:13.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:13.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:14.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:15.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:15.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:15.829 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:05:15.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:05:15.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:15.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:16.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:17.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:17.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:17.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:05:17.686 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:05:17.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:17.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:18.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:19.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:19.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:19.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:05:19.479 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:05:19.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:19.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:19.949 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3116284991,v1:127.0.0.1:6811/3116284991] [v2:127.0.0.1:6812/3116284991,v1:127.0.0.1:6813/3116284991] exists,up f9139018-e36b-4b94-90fd-d11736267f71 2024-04-25T09:05:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:05:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:05:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:05:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:05:19.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:05:19.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:05:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:05:19.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:05:19.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:05:19.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:05:19.954 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:05:19.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:05:19.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:05:19.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:05:19.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:05:19.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:05:19.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:05:19.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41cfbbc2-822d-4715-90b8-c021c8563066 2024-04-25T09:05:19.957 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 41cfbbc2-822d-4715-90b8-c021c8563066 2024-04-25T09:05:19.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 41cfbbc2-822d-4715-90b8-c021c8563066' 2024-04-25T09:05:19.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:05:19.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPHCpmmIsxOxAAla4Y8OQ4aUlYzR2p/A709A== 2024-04-25T09:05:19.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPHCpmmIsxOxAAla4Y8OQ4aUlYzR2p/A709A=="}' 2024-04-25T09:05:19.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41cfbbc2-822d-4715-90b8-c021c8563066 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:05:20.938 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:05:20.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:05:20.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPHCpmmIsxOxAAla4Y8OQ4aUlYzR2p/A709A== --osd-uuid 41cfbbc2-822d-4715-90b8-c021c8563066 2024-04-25T09:05:20.990 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:20.999+0000 7f147a9577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:21.044 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:21.055+0000 7f147a9577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:21.045 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:21.055+0000 7f147a9577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:21.046 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:21.055+0000 7f147a9577c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:05:21.046 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:21.055+0000 7f147a9577c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:05:21.046 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:21.055+0000 7f147a9577c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:05:21.144 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:21.155+0000 7f147a9577c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:05:27.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:05:27.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:05:27.457 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:05:27.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:05:27.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:05:28.021 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:05:28.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:05:28.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:05:28.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:05:28.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:05:28.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:05:28.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:28.067+0000 7f14102247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:28.069 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:28.079+0000 7f14102247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:28.070 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:28.083+0000 7f14102247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:28.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:05:28.476 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:05:28.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:28.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:28.639 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:28.651+0000 7f14102247c0 -1 Falling back to public interface 2024-04-25T09:05:28.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:29.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:29.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:29.980 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:05:29.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:05:29.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:29.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:30.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:31.111 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:05:31.123+0000 7f14102247c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:05:31.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:31.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:31.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:05:31.466 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:05:31.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:31.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:31.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:32.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:32.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:32.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:05:32.950 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:05:32.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:32.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:33.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:34.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:34.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:34.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:05:34.393 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:05:34.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:34.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:34.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:35.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:35.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:35.882 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:05:35.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:05:35.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:35.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:36.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:37.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:37.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:37.431 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:05:37.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:05:37.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:37.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:37.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:38.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:38.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:38.982 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:05:38.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:05:38.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:38.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:39.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:40.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:40.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:40.429 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:05:40.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:05:40.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:40.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:40.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:41.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:41.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:41.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:05:41.864 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:05:41.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:41.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:42.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:43.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:43.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:43.315 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:05:43.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:05:43.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:43.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:44.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:45.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:45.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:45.037 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:05:45.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:05:45.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:45.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:45.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:46.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:46.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:46.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:05:46.744 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:05:46.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:46.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:47.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:48.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:48.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:48.646 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:05:48.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:05:48.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:48.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:49.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:50.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:50.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:50.446 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:05:50.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:05:50.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:50.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:51.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:52.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:52.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:52.309 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:05:52.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:05:52.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:52.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:53.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:54.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:54.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:54.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:05:54.471 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:05:54.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:54.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:54.981 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 21 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1941426773,v1:127.0.0.1:6819/1941426773] [v2:127.0.0.1:6820/1941426773,v1:127.0.0.1:6821/1941426773] exists,up 41cfbbc2-822d-4715-90b8-c021c8563066 2024-04-25T09:05:54.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:05:54.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:05:54.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:05:54.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:05:54.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:54.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:54.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:54.983 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:54.984 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:54.984 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:54.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:54.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:54.984 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:55.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:55.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:55.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:55.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:55.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:55.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:55.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:55.523 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:05:55.523 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:05:55.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:55.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:55.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:55.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803793 2024-04-25T09:05:55.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803793 2024-04-25T09:05:55.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803793' 2024-04-25T09:05:55.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:55.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574860 2024-04-25T09:05:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574860 2024-04-25T09:05:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803793 1-55834574860' 2024-04-25T09:05:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:55.884 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313223 2024-04-25T09:05:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313223 2024-04-25T09:05:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803793 1-55834574860 2-90194313223' 2024-04-25T09:05:56.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:56.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:56.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803793 2024-04-25T09:05:56.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:56.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803793 2024-04-25T09:05:56.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:56.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803793 2024-04-25T09:05:56.033 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2024-04-25T09:05:56.033 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803793 2024-04-25T09:05:56.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:56.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803794 -lt 25769803793 2024-04-25T09:05:56.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:56.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574860 2024-04-25T09:05:56.498 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:56.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:56.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574860 2024-04-25T09:05:56.501 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:56.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574860 2024-04-25T09:05:56.502 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574860 2024-04-25T09:05:56.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574860' 2024-04-25T09:05:56.503 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:56.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574860 -lt 55834574860 2024-04-25T09:05:56.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:56.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313223 2024-04-25T09:05:56.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:56.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:05:56.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313223 2024-04-25T09:05:56.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:56.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313223 2024-04-25T09:05:56.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313223' 2024-04-25T09:05:56.943 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 90194313223 2024-04-25T09:05:56.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:57.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313223 -lt 90194313223 2024-04-25T09:05:57.371 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:05:57.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:57.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:57.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:58.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:58.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:58.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:58.351 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:58.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:05:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:05:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:05:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-04-25T09:05:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-04-25T09:05:58.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:05:59.409 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test-pool' already exists 2024-04-25T09:05:59.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:06:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T09:06:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:06:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:06:00.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:06:00.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:06:00.428 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:06:00.428 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:06:00.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:06:00.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:06:00.429 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:06:00.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:06:00.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:06:00.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:06:00.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:06:00.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:06:00.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:06:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:06:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:06:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:06:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:06:01.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:01.006 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:06:01.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803797 2024-04-25T09:06:01.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803797 2024-04-25T09:06:01.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803797' 2024-04-25T09:06:01.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:01.170 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:06:01.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574863 2024-04-25T09:06:01.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574863 2024-04-25T09:06:01.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803797 1-55834574863' 2024-04-25T09:06:01.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:01.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313226 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313226 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803797 1-55834574863 2-90194313226' 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803797 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:01.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:06:01.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803797 2024-04-25T09:06:01.488 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:01.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803797 2024-04-25T09:06:01.490 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803797 2024-04-25T09:06:01.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2024-04-25T09:06:01.490 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:06:01.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803794 -lt 25769803797 2024-04-25T09:06:01.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:06:02.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:06:02.905 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:06:03.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803797 -lt 25769803797 2024-04-25T09:06:03.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:03.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574863 2024-04-25T09:06:03.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:03.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:06:03.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574863 2024-04-25T09:06:03.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:03.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574863 2024-04-25T09:06:03.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574863' 2024-04-25T09:06:03.346 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574863 2024-04-25T09:06:03.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:06:03.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574863 -lt 55834574863 2024-04-25T09:06:03.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:03.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313226 2024-04-25T09:06:03.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:03.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:06:03.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313226 2024-04-25T09:06:03.766 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:03.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313226 2024-04-25T09:06:03.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313226' 2024-04-25T09:06:03.768 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 90194313226 2024-04-25T09:06:03.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:06:04.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313226 -lt 90194313226 2024-04-25T09:06:04.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:06:04.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:04.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:04.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:06:04.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:04.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:04.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:04.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:04.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:04.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:04.780 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:05.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:06:05.221 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:05.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:05.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:05.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:06:05.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:06:05.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:06:05.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:06:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:06:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.740011+0000 0'0 26:36 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:45:43.221041+0000 0 0 2024-04-25T09:06:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.740245+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:45:24.421779+0000 0 0 2024-04-25T09:06:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:55.452175+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:42:09.609577+0000 0 0 2024-04-25T09:06:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.741438+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:14:12.367905+0000 0 0 2024-04-25T09:06:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:06:00.492599+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:05:59.281024+0000 0'0 2024-04-25T09:05:59.281024+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:26:56.048563+0000 0 0 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.739829+0000 0'0 26:36 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:55:08.699591+0000 0 0 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:55.383871+0000 0'0 26:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:56:16.127177+0000 0 0 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:55.451911+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:17:48.751475+0000 0 0 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.740317+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:32:27.746602+0000 0 0 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:06:06.164 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:06:06.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:06:06.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:06:06.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:06:06.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:06.338 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-04-25T09:06:06.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:06:06.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:06:06.339 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:06:06.339 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:06.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:06.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:06.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:07.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:07.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-04-25T09:06:07.871 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T09:06:07.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-04-25T09:06:08.512 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T09:06:08.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:06:08.965 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:06:08.965 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.740011+0000 0'0 26:36 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:45:43.221041+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.740245+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:45:24.421779+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:55.452175+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:42:09.609577+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.741438+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:14:12.367905+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:06:00.492599+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:05:59.281024+0000 0'0 2024-04-25T09:05:59.281024+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:26:56.048563+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.739829+0000 0'0 26:36 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:55:08.699591+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-04-25T09:05:55.383871+0000 0'0 26:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:56:16.127177+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:55.451911+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:17:48.751475+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:05:54.740317+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:32:27.746602+0000 0 0 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:06:08.966 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T09:06:08.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:06:08.984 INFO:tasks.workunit.client.0.smithi031.stderr:4+0 records in 2024-04-25T09:06:08.984 INFO:tasks.workunit.client.0.smithi031.stderr:4+0 records out 2024-04-25T09:06:08.984 INFO:tasks.workunit.client.0.smithi031.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000184563 s, 22.2 MB/s 2024-04-25T09:06:08.985 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T09:06:08.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:08.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:09.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:09.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:10.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:10.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:10.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:10.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:10.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:10.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:10.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:10.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:10.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:10.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:10.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:11.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:11.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:12.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:12.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:13.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:13.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:14.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:14.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:15.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:15.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:16.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:16.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:17.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:17.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:18.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:18.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:19.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:19.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:20.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:20.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:21.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:21.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:22.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:22.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:23.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:23.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:24.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:24.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:25.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:26.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:27.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:27.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:28.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:28.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:29.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:29.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.228 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:30.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:30.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:31.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:31.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:32.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:32.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:33.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:33.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:34.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:34.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:35.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:35.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:35.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:35.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:35.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:35.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:35.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:35.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:35.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:35.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:36.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:36.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:36.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:36.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:36.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:36.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:36.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:36.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:36.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:36.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:36.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:36.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:37.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:37.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:38.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:38.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:38.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:38.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:38.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:38.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:38.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:38.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:38.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:38.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:38.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:38.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:39.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:39.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:40.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:40.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:41.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:41.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:42.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:42.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:43.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:43.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:44.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:44.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:45.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:45.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:46.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:46.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:47.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:47.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:48.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:48.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:49.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:49.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:50.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:50.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:51.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:51.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:51.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:06:51.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:06:51.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-04-25T09:06:51.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:06:51.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:06:51.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T09:06:51.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:06:51.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T09:06:51.441 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T09:06:51.441 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:06:51.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:06:51.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:06:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:06:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:06:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:06:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:06:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:06:51.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.2.asok 2024-04-25T09:06:51.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:06:52.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:06:52.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:06:52.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:06:52.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:06:52.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:06:52.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:06:52.056 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:52.056 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:52.057 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:06:52.057 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.2.asok 2024-04-25T09:06:52.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:06:52.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.2.asok injectdataerr test-pool obj250 2024-04-25T09:06:52.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:06:52.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:06:52.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T09:06:52.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:06:52.171 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T09:06:52.171 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T09:06:52.171 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:06:52.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:06:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T09:06:52.613 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:06:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:06:52.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:06:52.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:06:52.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:06:52.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:06:52.614 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:06:52.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:06:52.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:06:52.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:06:52.616 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:06:52.617 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:52.617 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:52.617 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:06:52.617 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:06:52.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:06:52.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:06:52.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:06:52.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:06:52.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:06:52.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:06:52.730 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:06:52.731 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:52.731 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:52.731 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:06:52.731 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:06:52.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:06:52.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr test-pool obj250 2024-04-25T09:06:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:06:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:06:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:06:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:06:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:06:52.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:06:52.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:52.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:52.856 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:52.856 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:52.856 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:06:52.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:06:52.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:06:52.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:06:52.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:06:52.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:06:52.858 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:06:52.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:06:52.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:06:52.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:06:52.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:06:52.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:06:52.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:06:52.891 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:06:52.901+0000 7fbc6759a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:52.893 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:06:52.901+0000 7fbc6759a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:52.895 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:06:52.905+0000 7fbc6759a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:53.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:06:53.329 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:06:53.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:53.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:53.440 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:06:53.449+0000 7fbc6759a7c0 -1 Falling back to public interface 2024-04-25T09:06:53.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:54.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:54.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:54.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:06:54.763 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:06:54.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:54.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:55.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:56.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:56.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:56.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:06:56.264 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:06:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:56.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:56.265 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:06:56.273+0000 7fbc6759a7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T09:06:56.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:57.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:57.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:57.730 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:06:57.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:06:57.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:57.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:58.373 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/1840477770,v1:127.0.0.1:6803/1840477770] [v2:127.0.0.1:6804/1840477770,v1:127.0.0.1:6805/1840477770] exists,up e6577b9e-d226-4e7a-b953-3f6b96e34685 2024-04-25T09:06:58.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:06:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:06:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:06:58.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-04-25T09:06:59.598 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. 2024-04-25T09:06:59.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-04-25T09:07:14.618 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-04-25T09:07:14.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:07:14.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:07:14.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:07:14.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:07:14.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:07:14.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:07:15.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:15.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:15.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:07:15.052 INFO:tasks.workunit.client.0.smithi031.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:15.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:07:15.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:07:15.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:07:16.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:07:16.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:07:16.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:07:16.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:07:16.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:07:16.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:07:16.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:16.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:07:16.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:16.501 INFO:tasks.workunit.client.0.smithi031.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:16.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:07:16.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:07:16.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:07:17.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:07:17.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:07:17.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:07:17.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:07:17.504 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:07:17.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:07:17.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:17.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:17.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:07:17.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:07:17.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:07:17.945 INFO:tasks.workunit.client.0.smithi031.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:17.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:07:18.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:07:18.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:07:18.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:07:18.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:07:18.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:07:18.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:07:19.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:19.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:19.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:07:19.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:07:19.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:07:19.380 INFO:tasks.workunit.client.0.smithi031.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:07:19.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:07:20.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:07:20.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:07:20.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:07:20.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:07:20.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:07:20.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:07:20.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:07:20.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:07:20.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:07:20.844 INFO:tasks.workunit.client.0.smithi031.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:07:20.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-04-25T09:07:20.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-04-25T09:07:20.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:07:21.248 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:07:21.248 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.419167+0000 0'0 41:78 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:15:08.389626+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.981413+0000 0'0 41:77 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:09:18.811199+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.957541+0000 0'0 41:61 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:08:29.520055+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.982165+0000 0'0 41:77 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:53:35.136215+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:07:16.030936+0000 30'301 41:901 [2,1,0] 2 [2,1] 2 0'0 2024-04-25T09:05:59.281024+0000 0'0 2024-04-25T09:05:59.281024+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.486237+0000 0'0 41:78 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:11:27.644924+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.981978+0000 0'0 41:68 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:44:25.436311+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.984967+0000 0'0 41:61 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:12:03.888841+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:07:00.981523+0000 0'0 41:77 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:04:13.603757+0000 0'0 2024-04-25T09:04:13.603757+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:19:35.033818+0000 0 0 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:07:21.249 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T09:07:21.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:07:21.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:26.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-04-25T09:07:26.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:07:26.688 INFO:tasks.workunit.client.0.smithi031.stderr:pg has no unfound objects 2024-04-25T09:07:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T09:07:26.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:07:26.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:07:26.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:07:26.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:07:26.707 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:07:26.707 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:07:26.708 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:07:26.708 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:07:26.708 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:07:26.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:07:26.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:07:26.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:07:26.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:07:26.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:07:26.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:07:27.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:07:27.266 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:07:27.266 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:07:27.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:07:27.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:27.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:07:27.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953480 2024-04-25T09:07:27.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953480 2024-04-25T09:07:27.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953480' 2024-04-25T09:07:27.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:27.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:07:27.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574882 2024-04-25T09:07:27.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574882 2024-04-25T09:07:27.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953480 1-55834574882' 2024-04-25T09:07:27.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:27.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:07:27.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313245 2024-04-25T09:07:27.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313245 2024-04-25T09:07:27.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953480 1-55834574882 2-90194313245' 2024-04-25T09:07:27.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:27.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-137438953480 2024-04-25T09:07:27.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:28.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:07:28.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-137438953480 2024-04-25T09:07:28.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:28.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953480 2024-04-25T09:07:28.002 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 137438953480 2024-04-25T09:07:28.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 137438953480' 2024-04-25T09:07:28.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:28.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953481 -lt 137438953480 2024-04-25T09:07:28.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:28.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574882 2024-04-25T09:07:28.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:28.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:07:28.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574882 2024-04-25T09:07:28.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:28.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574882 2024-04-25T09:07:28.486 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 55834574882 2024-04-25T09:07:28.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574882' 2024-04-25T09:07:28.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:07:29.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574882 -lt 55834574882 2024-04-25T09:07:29.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:29.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313245 2024-04-25T09:07:29.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:29.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:07:29.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313245 2024-04-25T09:07:29.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:29.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313245 2024-04-25T09:07:29.134 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 90194313245 2024-04-25T09:07:29.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313245' 2024-04-25T09:07:29.134 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:07:29.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313245 -lt 90194313245 2024-04-25T09:07:29.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:07:29.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:29.617 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:30.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:07:30.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:07:30.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:07:30.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:07:30.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:07:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:07:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:07:30.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:07:30.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:07:30.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:07:30.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:30.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:31.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:07:31.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:07:31.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:07:31.270 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T09:07:31.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-04-25T09:07:31.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-04-25T09:07:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-04-25T09:07:31.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-04-25T09:07:31.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-04-25T09:07:31.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-04-25T09:07:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-04-25T09:07:31.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-04-25T09:07:31.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-04-25T09:07:31.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-04-25T09:07:31.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-04-25T09:07:31.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-04-25T09:07:31.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-04-25T09:07:31.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-04-25T09:07:31.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-04-25T09:07:31.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-04-25T09:07:31.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-04-25T09:07:31.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:31.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:31.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-04-25T09:07:31.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-04-25T09:07:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-04-25T09:07:32.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-04-25T09:07:32.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-04-25T09:07:32.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-04-25T09:07:32.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-04-25T09:07:32.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-04-25T09:07:32.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-04-25T09:07:32.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-04-25T09:07:32.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-04-25T09:07:32.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-04-25T09:07:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-04-25T09:07:32.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-04-25T09:07:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-04-25T09:07:32.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-04-25T09:07:32.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-04-25T09:07:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-04-25T09:07:32.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-04-25T09:07:32.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-04-25T09:07:32.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-04-25T09:07:32.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-04-25T09:07:32.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-04-25T09:07:32.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-04-25T09:07:32.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-04-25T09:07:32.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-04-25T09:07:32.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-04-25T09:07:32.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-04-25T09:07:32.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-04-25T09:07:32.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-04-25T09:07:32.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:32.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-04-25T09:07:32.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:32.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-04-25T09:07:33.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-04-25T09:07:33.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-04-25T09:07:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-04-25T09:07:33.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-04-25T09:07:33.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-04-25T09:07:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-04-25T09:07:33.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-04-25T09:07:33.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-04-25T09:07:33.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-04-25T09:07:33.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-04-25T09:07:33.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-04-25T09:07:33.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-04-25T09:07:33.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-04-25T09:07:33.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-04-25T09:07:33.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-04-25T09:07:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-04-25T09:07:33.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-04-25T09:07:33.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-04-25T09:07:33.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-04-25T09:07:33.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-04-25T09:07:33.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-04-25T09:07:33.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-04-25T09:07:33.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-04-25T09:07:33.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-04-25T09:07:33.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-04-25T09:07:33.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-04-25T09:07:33.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-04-25T09:07:33.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-04-25T09:07:33.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-04-25T09:07:33.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-04-25T09:07:33.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-04-25T09:07:33.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:33.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:33.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-04-25T09:07:33.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-04-25T09:07:34.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-04-25T09:07:34.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-04-25T09:07:34.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-04-25T09:07:34.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-04-25T09:07:34.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-04-25T09:07:34.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-04-25T09:07:34.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-04-25T09:07:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-04-25T09:07:34.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-04-25T09:07:34.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-04-25T09:07:34.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-04-25T09:07:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-04-25T09:07:34.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-04-25T09:07:34.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-04-25T09:07:34.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-04-25T09:07:34.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-04-25T09:07:34.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-04-25T09:07:34.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-04-25T09:07:34.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-04-25T09:07:34.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-04-25T09:07:34.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-04-25T09:07:34.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-04-25T09:07:34.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-04-25T09:07:34.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-04-25T09:07:34.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-04-25T09:07:34.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-04-25T09:07:34.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-04-25T09:07:34.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-04-25T09:07:34.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-04-25T09:07:34.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-04-25T09:07:34.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-04-25T09:07:34.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-04-25T09:07:35.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-04-25T09:07:35.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-04-25T09:07:35.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-04-25T09:07:35.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-04-25T09:07:35.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-04-25T09:07:35.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-04-25T09:07:35.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-04-25T09:07:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-04-25T09:07:35.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.276 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-04-25T09:07:35.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-04-25T09:07:35.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-04-25T09:07:35.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-04-25T09:07:35.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-04-25T09:07:35.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-04-25T09:07:35.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-04-25T09:07:35.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-04-25T09:07:35.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-04-25T09:07:35.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-04-25T09:07:35.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.581 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-04-25T09:07:35.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-04-25T09:07:35.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-04-25T09:07:35.642 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-04-25T09:07:35.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-04-25T09:07:35.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-04-25T09:07:35.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-04-25T09:07:35.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-04-25T09:07:35.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-04-25T09:07:35.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-04-25T09:07:35.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-04-25T09:07:35.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-04-25T09:07:35.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-04-25T09:07:35.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:35.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:35.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-04-25T09:07:35.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-04-25T09:07:36.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-04-25T09:07:36.037 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-04-25T09:07:36.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-04-25T09:07:36.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-04-25T09:07:36.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-04-25T09:07:36.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-04-25T09:07:36.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-04-25T09:07:36.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-04-25T09:07:36.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-04-25T09:07:36.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-04-25T09:07:36.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-04-25T09:07:36.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-04-25T09:07:36.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-04-25T09:07:36.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-04-25T09:07:36.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-04-25T09:07:36.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-04-25T09:07:36.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-04-25T09:07:36.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-04-25T09:07:36.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-04-25T09:07:36.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-04-25T09:07:36.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-04-25T09:07:36.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-04-25T09:07:36.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-04-25T09:07:36.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-04-25T09:07:36.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-04-25T09:07:36.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-04-25T09:07:36.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-04-25T09:07:36.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-04-25T09:07:36.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-04-25T09:07:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:36.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:36.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-04-25T09:07:36.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-04-25T09:07:37.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-04-25T09:07:37.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-04-25T09:07:37.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-04-25T09:07:37.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-04-25T09:07:37.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-04-25T09:07:37.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-04-25T09:07:37.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-04-25T09:07:37.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-04-25T09:07:37.261 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-04-25T09:07:37.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-04-25T09:07:37.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-04-25T09:07:37.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-04-25T09:07:37.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-04-25T09:07:37.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-04-25T09:07:37.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-04-25T09:07:37.473 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-04-25T09:07:37.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-04-25T09:07:37.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-04-25T09:07:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-04-25T09:07:37.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-04-25T09:07:37.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-04-25T09:07:37.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-04-25T09:07:37.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-04-25T09:07:37.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-04-25T09:07:37.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-04-25T09:07:37.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-04-25T09:07:37.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-04-25T09:07:37.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-04-25T09:07:37.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-04-25T09:07:37.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-04-25T09:07:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:37.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:37.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-04-25T09:07:37.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-04-25T09:07:38.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-04-25T09:07:38.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-04-25T09:07:38.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-04-25T09:07:38.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-04-25T09:07:38.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-04-25T09:07:38.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-04-25T09:07:38.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-04-25T09:07:38.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-04-25T09:07:38.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-04-25T09:07:38.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-04-25T09:07:38.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-04-25T09:07:38.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-04-25T09:07:38.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-04-25T09:07:38.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-04-25T09:07:38.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-04-25T09:07:38.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-04-25T09:07:38.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-04-25T09:07:38.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-04-25T09:07:38.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-04-25T09:07:38.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-04-25T09:07:38.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-04-25T09:07:38.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-04-25T09:07:38.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-04-25T09:07:38.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-04-25T09:07:38.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-04-25T09:07:38.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-04-25T09:07:38.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:38.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:38.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-04-25T09:07:38.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-04-25T09:07:39.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-04-25T09:07:39.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-04-25T09:07:39.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-04-25T09:07:39.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-04-25T09:07:39.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-04-25T09:07:39.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-04-25T09:07:39.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-04-25T09:07:39.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-04-25T09:07:39.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-04-25T09:07:39.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-04-25T09:07:39.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-04-25T09:07:39.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-04-25T09:07:39.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.403 INFO:tasks.workunit.client.0.smithi031.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-04-25T09:07:39.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-04-25T09:07:39.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-04-25T09:07:39.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-04-25T09:07:39.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-04-25T09:07:39.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-04-25T09:07:39.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-04-25T09:07:39.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-04-25T09:07:39.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-04-25T09:07:39.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-04-25T09:07:39.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-04-25T09:07:39.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-04-25T09:07:39.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-04-25T09:07:39.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-04-25T09:07:39.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-04-25T09:07:39.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-04-25T09:07:39.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-04-25T09:07:39.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-04-25T09:07:39.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-04-25T09:07:39.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-04-25T09:07:39.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:39.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:39.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-04-25T09:07:39.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-04-25T09:07:40.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-04-25T09:07:40.044 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-04-25T09:07:40.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-04-25T09:07:40.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-04-25T09:07:40.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-04-25T09:07:40.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-04-25T09:07:40.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-04-25T09:07:40.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-04-25T09:07:40.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-04-25T09:07:40.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-04-25T09:07:40.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-04-25T09:07:40.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-04-25T09:07:40.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.429 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-04-25T09:07:40.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-04-25T09:07:40.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-04-25T09:07:40.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-04-25T09:07:40.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-04-25T09:07:40.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-04-25T09:07:40.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-04-25T09:07:40.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-04-25T09:07:40.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-04-25T09:07:40.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-04-25T09:07:40.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-04-25T09:07:40.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-04-25T09:07:40.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-04-25T09:07:40.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-04-25T09:07:40.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-04-25T09:07:40.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-04-25T09:07:40.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:40.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:07:40.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-04-25T09:07:40.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:41.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:41.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:07:41.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-04-25T09:07:41.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:07:41.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:07:41.517 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test-pool' does not exist 2024-04-25T09:07:41.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:07:41.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:07:41.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:07:41.535 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:07:41.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:07:41.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:07:41.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:07:41.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:07:41.848 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:07:41.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:07:41.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:07:41.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:07:41.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:07:41.851 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:07:41.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:41.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:07:41.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:07:41.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:41.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:07:41.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:07:41.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:07:41.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:41.900 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:41.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:07:41.902 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:07:41.903 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:07:41.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:07:41.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:07:41.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:07:41.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:07:41.905 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:07:41.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:07:41.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:07:41.911 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:41.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:07:41.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:07:41.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:07:41.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:41.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:41.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:07:41.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:07:41.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:07:41.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:07:41.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:41.917 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:41.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.383712 2024-04-25T09:07:41.919 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:07:41.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:07:41.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:07:41.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:07:41.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:07:42.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:07:42.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:42.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:42.383 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:42.383 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:42.383 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:42.384 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:07:42.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:07:42.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:07:42.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:07:42.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:07:42.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:07:42.596 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:07:42.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:07:42.597 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:07:42.597 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:07:42.598 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:07:42.598 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:07:42.599 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:42.599 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:42.599 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T09:07:42.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:07:42.599 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get fsid 2024-04-25T09:07:42.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:07:42.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:07:42.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:07:42.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:07:42.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-mon.a.asok 2024-04-25T09:07:42.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:07:42.719 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.383712/ceph-mon.a.asok config get mon_host 2024-04-25T09:07:42.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:07:42.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:07:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:07:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:07:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:07:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:07:43.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:07:43.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:07:43.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:07:44.163 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T09:07:44.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:07:44.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:07:44.183 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T09:07:44.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T09:07:44.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:07:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:07:44.189 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:07:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:07:44.191 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:07:44.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6f4c7e73-f1a7-49ee-b89f-8a6441440c8c 2024-04-25T09:07:44.191 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 6f4c7e73-f1a7-49ee-b89f-8a6441440c8c 2024-04-25T09:07:44.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6f4c7e73-f1a7-49ee-b89f-8a6441440c8c' 2024-04-25T09:07:44.192 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:07:44.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgHSpmRmpjDRAAQXKFFlPpwhWJTJBxGbgvqA== 2024-04-25T09:07:44.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgHSpmRmpjDRAAQXKFFlPpwhWJTJBxGbgvqA=="}' 2024-04-25T09:07:44.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6f4c7e73-f1a7-49ee-b89f-8a6441440c8c -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:07:44.525 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:07:44.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:07:44.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgHSpmRmpjDRAAQXKFFlPpwhWJTJBxGbgvqA== --osd-uuid 6f4c7e73-f1a7-49ee-b89f-8a6441440c8c 2024-04-25T09:07:44.571 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.581+0000 7f868d4017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:44.591 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.601+0000 7f868d4017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:44.593 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.601+0000 7f868d4017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:44.593 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.601+0000 7f868d4017c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:07:44.593 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.601+0000 7f868d4017c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:07:44.593 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.601+0000 7f868d4017c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:07:44.631 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:44.641+0000 7f868d4017c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:07:50.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:07:50.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:07:50.403 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:07:50.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:07:50.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:07:51.030 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:07:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:07:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:07:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:07:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:07:51.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:07:51.066 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:51.076+0000 7fc8eb7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:51.087 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:51.096+0000 7fc8eb7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:51.089 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:51.096+0000 7fc8eb7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:51.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:07:51.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:07:51.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:07:51.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:07:51.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:07:51.469 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:07:51.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:51.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:07:51.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:51.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:51.652 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:51.660+0000 7fc8eb7927c0 -1 Falling back to public interface 2024-04-25T09:07:51.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:52.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:52.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:52.906 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:07:52.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:07:52.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:52.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:53.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:54.141 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:07:54.148+0000 7fc8eb7927c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:07:54.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:54.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:54.350 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:07:54.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:07:54.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:54.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:54.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:55.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:55.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:55.883 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:07:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:07:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:55.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:56.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:57.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:57.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:57.336 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:07:57.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:07:57.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:57.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:57.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:58.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:58.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:58.868 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:07:58.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:07:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:59.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:00.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:00.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:00.308 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:08:00.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:08:00.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:00.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:00.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:01.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:01.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:01.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:08:01.883 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:08:01.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:01.884 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:02.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:03.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:03.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:03.319 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:08:03.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:08:03.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:03.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:03.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:04.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:04.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:04.904 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:08:04.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:08:04.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:04.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:05.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:06.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:06.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:06.486 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:08:06.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:08:06.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:06.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:07.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:08.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:08.110 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:08:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:08:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:08.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:08.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:09.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:09.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:09.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:08:09.698 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:08:09.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:09.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:10.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:11.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:11.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:11.568 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:08:11.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:08:11.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:11.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:12.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:13.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:13.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:13.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:08:13.349 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:08:13.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:13.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:14.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:15.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:15.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:15.078 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:08:15.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:08:15.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:15.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:15.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:16.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:16.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:16.920 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:08:16.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:08:16.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:16.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:17.457 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2080393651,v1:127.0.0.1:6803/2080393651] [v2:127.0.0.1:6804/2080393651,v1:127.0.0.1:6805/2080393651] exists,up 6f4c7e73-f1a7-49ee-b89f-8a6441440c8c 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:08:17.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:08:17.461 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:08:17.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:08:17.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:08:17.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:08:17.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:08:17.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1868b8cd-1ef9-44db-8653-7093a7c81dff 2024-04-25T09:08:17.465 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 1868b8cd-1ef9-44db-8653-7093a7c81dff 2024-04-25T09:08:17.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1868b8cd-1ef9-44db-8653-7093a7c81dff' 2024-04-25T09:08:17.465 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:08:17.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBHSpm5Pn4HRAA/jJUaCMmsstiVHI1BVvFyg== 2024-04-25T09:08:17.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBHSpm5Pn4HRAA/jJUaCMmsstiVHI1BVvFyg=="}' 2024-04-25T09:08:17.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1868b8cd-1ef9-44db-8653-7093a7c81dff -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:08:17.998 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:08:18.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:08:18.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBHSpm5Pn4HRAA/jJUaCMmsstiVHI1BVvFyg== --osd-uuid 1868b8cd-1ef9-44db-8653-7093a7c81dff 2024-04-25T09:08:18.053 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.060+0000 7f61625d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:18.055 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.064+0000 7f61625d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:18.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.064+0000 7f61625d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:18.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.064+0000 7f61625d67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:08:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.064+0000 7f61625d67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:08:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.064+0000 7f61625d67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:08:18.098 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:18.108+0000 7f61625d67c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:08:24.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:08:24.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:08:24.252 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:08:24.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:08:24.252 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:08:24.840 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:08:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:08:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:08:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:08:24.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:08:24.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:08:24.876 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:24.884+0000 7fcf431d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:24.878 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:24.888+0000 7fcf431d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:24.879 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:24.888+0000 7fcf431d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:25.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:08:25.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:08:25.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:08:25.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:08:25.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:08:25.270 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:08:25.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:25.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:08:25.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:25.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:25.451 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:25.460+0000 7fcf431d57c0 -1 Falling back to public interface 2024-04-25T09:08:25.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:26.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:26.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:26.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:08:26.717 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:08:26.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:26.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:27.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:27.995 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:28.008+0000 7fcf431d57c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:08:28.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:28.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:28.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:08:28.174 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:08:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:28.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:29.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:29.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:29.629 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:08:29.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:08:29.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:29.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:30.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:31.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:31.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:31.063 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:08:31.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:08:31.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:31.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:31.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:32.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:32.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:32.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:08:32.613 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:08:32.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:32.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:33.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:34.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:34.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:34.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:08:34.147 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:08:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:34.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:35.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:35.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:35.606 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:08:35.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:08:35.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:35.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:36.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:37.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:37.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:37.072 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:08:37.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:08:37.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:37.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:37.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:38.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:38.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:38.528 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:08:38.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:08:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:38.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:39.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:40.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:40.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:40.266 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:08:40.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:08:40.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:40.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:40.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:41.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:41.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:41.768 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:08:41.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:08:41.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:41.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:42.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:43.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:43.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:43.453 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:08:43.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:08:43.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:43.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:44.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:45.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:45.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:45.412 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:08:45.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:08:45.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:45.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:46.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:08:47.272 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:08:47.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:47.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:48.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:49.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:49.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:49.095 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:08:49.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:08:49.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:49.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:49.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:50.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:50.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:50.959 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:08:50.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:08:50.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:50.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:51.405 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4162894629,v1:127.0.0.1:6811/4162894629] [v2:127.0.0.1:6812/4162894629,v1:127.0.0.1:6813/4162894629] exists,up 1868b8cd-1ef9-44db-8653-7093a7c81dff 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:08:51.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:08:51.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:08:51.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:08:51.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:08:51.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3405b48-ce47-4686-a36f-018a33648b67 2024-04-25T09:08:51.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f3405b48-ce47-4686-a36f-018a33648b67' 2024-04-25T09:08:51.412 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 f3405b48-ce47-4686-a36f-018a33648b67 2024-04-25T09:08:51.412 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:08:51.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjHSpmJIyMGhAAOVv/VPrknSUeb2yr/vBy0w== 2024-04-25T09:08:51.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjHSpmJIyMGhAAOVv/VPrknSUeb2yr/vBy0w=="}' 2024-04-25T09:08:51.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3405b48-ce47-4686-a36f-018a33648b67 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:08:52.089 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:08:52.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:08:52.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjHSpmJIyMGhAAOVv/VPrknSUeb2yr/vBy0w== --osd-uuid f3405b48-ce47-4686-a36f-018a33648b67 2024-04-25T09:08:52.149 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.159+0000 7fa87a7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:52.206 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.219+0000 7fa87a7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:52.209 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.219+0000 7fa87a7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:52.210 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.219+0000 7fa87a7927c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:08:52.210 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.219+0000 7fa87a7927c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:08:52.210 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.219+0000 7fa87a7927c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:08:52.339 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:52.351+0000 7fa87a7927c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:08:58.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:08:58.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:08:58.874 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:08:58.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:08:58.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:08:59.451 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:08:59.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:08:59.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:08:59.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:08:59.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:08:59.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:08:59.486 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:59.499+0000 7f89f14a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:59.515 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:59.527+0000 7f89f14a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:59.517 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:08:59.527+0000 7f89f14a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:59.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:08:59.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:08:59.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:08:59.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:08:59.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:08:59.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:59.892 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:08:59.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:08:59.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:59.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:00.099 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:09:00.111+0000 7f89f14a37c0 -1 Falling back to public interface 2024-04-25T09:09:00.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:01.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:01.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:01.345 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:09:01.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:09:01.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:01.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:01.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:02.673 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:09:02.683+0000 7f89f14a37c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:09:02.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:02.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:02.777 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:09:02.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:09:02.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:02.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:03.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:04.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:04.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:04.355 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:09:04.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:09:04.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:04.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:04.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:05.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:05.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:05.925 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:09:05.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:09:05.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:05.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:06.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:07.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:07.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:07.485 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:09:07.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:09:07.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:07.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:07.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:08.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:08.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:08.933 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:09:08.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:09:08.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:08.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:09.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:10.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:10.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:10.489 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:09:10.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:09:10.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:10.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:10.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:11.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:11.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:11.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:09:11.971 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:09:11.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:11.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:12.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:13.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:13.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:13.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:09:13.552 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:09:13.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:13.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:13.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:14.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:14.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:14.997 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:09:14.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:09:14.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:14.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:15.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:16.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:16.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:16.845 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:09:16.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:09:16.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:16.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:17.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:18.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:18.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:18.863 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:09:18.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:09:18.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:18.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:20.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:21.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:21.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:21.232 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:09:21.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:09:21.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:21.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:22.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:23.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:23.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:23.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:09:23.259 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:09:23.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:23.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:24.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:25.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:25.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:25.242 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:09:25.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:09:25.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:25.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:25.715 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3741855117,v1:127.0.0.1:6819/3741855117] [v2:127.0.0.1:6820/3741855117,v1:127.0.0.1:6821/3741855117] exists,up f3405b48-ce47-4686-a36f-018a33648b67 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:09:25.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:09:25.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:09:25.717 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:09:25.717 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:09:25.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:09:25.717 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:09:25.718 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:09:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:09:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:09:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:09:25.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:09:25.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:09:25.830 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:09:26.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:09:26.400 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:09:26.401 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:09:26.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:09:26.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:26.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:09:26.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803801 2024-04-25T09:09:26.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803801 2024-04-25T09:09:26.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801' 2024-04-25T09:09:26.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:26.632 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:09:26.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542159 2024-04-25T09:09:26.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542159 2024-04-25T09:09:26.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-60129542159' 2024-04-25T09:09:26.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:26.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:09:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=94489280517 2024-04-25T09:09:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 94489280517 2024-04-25T09:09:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-60129542159 2-94489280517' 2024-04-25T09:09:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:26.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803801 2024-04-25T09:09:26.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:26.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:09:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803801 2024-04-25T09:09:26.994 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:26.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803801 2024-04-25T09:09:26.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-04-25T09:09:26.996 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803801 2024-04-25T09:09:26.996 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:27.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803799 -lt 25769803801 2024-04-25T09:09:27.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:09:28.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:09:28.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:28.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803801 -lt 25769803801 2024-04-25T09:09:28.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:28.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-60129542159 2024-04-25T09:09:28.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:28.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:09:28.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-60129542159 2024-04-25T09:09:28.886 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:28.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542159 2024-04-25T09:09:28.888 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 60129542159 2024-04-25T09:09:28.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 60129542159' 2024-04-25T09:09:28.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:09:29.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542159 -lt 60129542159 2024-04-25T09:09:29.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:29.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-94489280517 2024-04-25T09:09:29.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:29.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:09:29.309 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-94489280517 2024-04-25T09:09:29.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:29.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=94489280517 2024-04-25T09:09:29.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 94489280517' 2024-04-25T09:09:29.311 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 94489280517 2024-04-25T09:09:29.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:09:29.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 94489280517 -lt 94489280517 2024-04-25T09:09:29.744 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:09:29.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:29.745 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:30.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:09:30.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:09:30.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:09:30.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:09:30.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:30.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:09:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:09:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:09:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-04-25T09:09:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-04-25T09:09:31.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:09:31.813 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test-pool' already exists 2024-04-25T09:09:31.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:09:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:09:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:09:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:09:32.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:09:32.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:09:32.832 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:09:32.832 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:09:32.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:09:32.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:09:32.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:09:32.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:09:32.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:09:32.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:09:32.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:09:32.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:09:32.965 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:09:33.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:09:33.450 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:09:33.451 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:09:33.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:09:33.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:33.451 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:09:33.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803804 2024-04-25T09:09:33.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803804 2024-04-25T09:09:33.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804' 2024-04-25T09:09:33.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:33.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:09:33.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542162 2024-04-25T09:09:33.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542162 2024-04-25T09:09:33.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-60129542162' 2024-04-25T09:09:33.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:33.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:09:33.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=94489280520 2024-04-25T09:09:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 94489280520 2024-04-25T09:09:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-60129542162 2-94489280520' 2024-04-25T09:09:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:33.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803804 2024-04-25T09:09:33.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:33.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:09:33.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:33.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803804 2024-04-25T09:09:33.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803804 2024-04-25T09:09:33.933 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 25769803804 2024-04-25T09:09:33.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-04-25T09:09:33.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:34.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803802 -lt 25769803804 2024-04-25T09:09:34.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:09:35.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:09:35.369 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:35.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803804 -lt 25769803804 2024-04-25T09:09:35.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:35.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-60129542162 2024-04-25T09:09:35.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:35.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:09:35.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-60129542162 2024-04-25T09:09:35.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:35.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542162 2024-04-25T09:09:35.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 60129542162' 2024-04-25T09:09:35.845 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 60129542162 2024-04-25T09:09:35.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:09:36.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542162 -lt 60129542162 2024-04-25T09:09:36.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:36.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-94489280520 2024-04-25T09:09:36.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:36.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:09:36.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-94489280520 2024-04-25T09:09:36.292 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:36.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=94489280520 2024-04-25T09:09:36.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 94489280520' 2024-04-25T09:09:36.293 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 94489280520 2024-04-25T09:09:36.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:09:36.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 94489280521 -lt 94489280520 2024-04-25T09:09:36.749 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:09:36.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:36.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:09:37.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:09:37.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:09:37.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:09:37.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:37.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:38.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:09:38.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:09:38.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:09:38.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:27.278621+0000 0'0 27:31 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:39:50.872226+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:27.362077+0000 0'0 27:31 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:54:19.635123+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:26.374372+0000 0'0 27:19 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:06:51.167699+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:27.278728+0000 0'0 27:31 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:49:28.551651+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:33.146936+0000 0'0 27:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:09:31.663876+0000 0'0 2024-04-25T09:09:31.663876+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:57:11.861782+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:27.362366+0000 0'0 27:31 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:40:06.536065+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:26.464785+0000 0'0 27:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:52:44.408091+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:26.463477+0000 0'0 27:19 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:19:59.256076+0000 0 0 2024-04-25T09:09:38.731 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:09:27.278216+0000 0'0 27:31 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:10:22.026371+0000 0 0 2024-04-25T09:09:38.732 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:09:38.732 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:09:38.732 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:09:38.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:09:38.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:09:38.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:09:38.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:09:38.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:38.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-04-25T09:09:38.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:09:38.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:09:38.900 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:09:38.900 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr:0' 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-04-25T09:09:39.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-04-25T09:09:39.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:09:39.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:09:39.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:09:39.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:09:39.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:09:39.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:09:40.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:09:40.650 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-04-25T09:09:41.283 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already down. 2024-04-25T09:09:41.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-04-25T09:09:41.877 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. 2024-04-25T09:09:41.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:09:42.306 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:09:42.306 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.415387+0000 0'0 29:38 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.414122+0000 0'0 29:38 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.777412+0000 0'0 30:27 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.414398+0000 0'0 29:38 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 active+undersized+degraded 2024-04-25T09:09:40.861503+0000 27'1 30:22 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:09:31.663876+0000 0'0 2024-04-25T09:09:31.663876+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.414132+0000 0'0 29:38 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.414474+0000 0'0 29:42 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:06:05.411641+0000 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.777387+0000 0'0 30:27 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-25T09:09:40.291681+0000 0'0 29:38 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:09:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:09:42.308 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T09:09:42.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:09:42.325 INFO:tasks.workunit.client.0.smithi031.stderr:4+0 records in 2024-04-25T09:09:42.325 INFO:tasks.workunit.client.0.smithi031.stderr:4+0 records out 2024-04-25T09:09:42.325 INFO:tasks.workunit.client.0.smithi031.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000282341 s, 14.5 MB/s 2024-04-25T09:09:42.326 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:09:42.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:42.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:42.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:42.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:42.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:42.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:42.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:42.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:42.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:43.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:43.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:44.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:44.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:45.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:46.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:46.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:47.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:47.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:48.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:48.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:49.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:49.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:50.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:50.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:51.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:51.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:52.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:52.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.559 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:53.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:53.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:54.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:54.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:55.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:55.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:55.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:55.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:55.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:55.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:55.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:55.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:55.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:55.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:56.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:56.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:56.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:56.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:56.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:56.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:56.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:56.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:56.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:09:56.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:09:56.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:09:56.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:09:56.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:09:56.760 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:09:56.760 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:09:57.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:09:57.241 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T09:09:57.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:09:57.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:09:57.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:09:57.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:09:57.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:09:57.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:09:57.242 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:09:57.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:09:57.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:09:57.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:09:57.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:09:57.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:09:57.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.2.asok 2024-04-25T09:09:57.245 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:09:57.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:09:57.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:09:57.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:09:57.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:09:57.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:09:57.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:09:57.365 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:09:57.365 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:57.366 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:09:57.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.2.asok 2024-04-25T09:09:57.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:09:57.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.2.asok injectdataerr test-pool obj75 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:09:57.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:09:57.483 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:09:57.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:09:57.484 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:09:57.484 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:09:57.484 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:09:57.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:09:57.943 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T09:09:57.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:09:57.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:09:57.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:09:57.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:09:57.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:09:57.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:09:57.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:09:57.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:57.946 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:09:57.947 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:09:57.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.383712/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:09:58.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:09:58.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:09:58.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:09:58.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:09:58.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:09:58.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:09:58.070 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:09:58.070 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:58.071 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:09:58.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.383712/ceph-osd.1.asok 2024-04-25T09:09:58.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:09:58.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.383712/ceph-osd.1.asok injectdataerr test-pool obj75 2024-04-25T09:09:58.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:09:58.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:09:58.190 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:09:58.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' 2024-04-25T09:09:58.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:09:58.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:09:58.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:09:58.192 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:09:58.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:09:58.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=2bb1c137-dcba-43b0-a004-7c88fee34ade --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.383712/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:09:58.193 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:09:58.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:09:58.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:09:58.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:09:58.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:09:58.227 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:09:58.238+0000 7f32c20b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:58.230 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:09:58.242+0000 7f32c20b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:58.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:09:58.242+0000 7f32c20b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:58.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:09:58.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:09:58.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:09:58.634 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:09:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:09:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:09:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:09:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:58.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:58.783 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:09:58.794+0000 7f32c20b67c0 -1 Falling back to public interface 2024-04-25T09:09:59.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:00.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:00.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:00.095 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:10:00.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:10:00.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:00.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:00.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:01.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:01.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:01.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:10:01.587 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:10:01.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:01.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:01.739 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:01.750+0000 7f32c20b67c0 -1 osd.0 27 log_to_monitors true 2024-04-25T09:10:02.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:03.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:03.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:03.133 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:10:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:10:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:03.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:03.788 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up out weight 0 up_from 33 up_thru 22 down_at 28 last_clean_interval [6,27) [v2:127.0.0.1:6802/2775549566,v1:127.0.0.1:6803/2775549566] [v2:127.0.0.1:6804/2775549566,v1:127.0.0.1:6805/2775549566] exists,up 6f4c7e73-f1a7-49ee-b89f-8a6441440c8c 2024-04-25T09:10:03.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:10:03.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:10:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:10:03.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-04-25T09:10:04.549 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. 2024-04-25T09:10:04.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-04-25T09:10:19.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:10:19.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-04-25T09:10:19.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-04-25T09:10:19.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:10:19.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:10:19.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:10:19.573 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:10:20.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-04-25T09:10:20.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-04-25T09:10:20.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-04-25T09:10:20.006 INFO:tasks.workunit.client.0.smithi031.stdout:active+recovery_unfound+degraded 2024-04-25T09:10:20.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-04-25T09:10:20.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-04-25T09:10:20.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:05.329347+0000 0'0 40:71 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:52:28.167267+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:06.478285+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:45:46.632368+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:06.479220+0000 0'0 40:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:36:27.150974+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:06.478342+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:34:12.675854+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-04-25T09:10:09.696069+0000 31'101 40:353 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:09:31.663876+0000 0'0 2024-04-25T09:09:31.663876+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:05.329229+0000 0'0 40:71 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:13:39.803207+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:05.916311+0000 0'0 40:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:51:40.628723+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:06.478561+0000 0'0 40:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:25:46.542828+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:10:06.480465+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:07:44.043044+0000 0'0 2024-04-25T09:07:44.043044+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:33:18.323515+0000 0 0 2024-04-25T09:10:20.409 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:10:20.410 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:10:20.410 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pgs 2024-04-25T09:10:20.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:10:20.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-04-25T09:10:20.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:25.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-04-25T09:10:25.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:10:25.828 INFO:tasks.workunit.client.0.smithi031.stderr:pg has no unfound objects 2024-04-25T09:10:25.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:10:25.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:10:25.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:10:25.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:10:25.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:10:25.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:10:25.850 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:10:25.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:10:25.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:10:25.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:10:25.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:10:25.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:10:25.974 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:10:26.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:10:26.453 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:10:26.453 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:10:26.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:10:26.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:26.453 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:10:26.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920775 2024-04-25T09:10:26.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920775 2024-04-25T09:10:26.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-141733920775' 2024-04-25T09:10:26.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:26.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:10:26.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542175 2024-04-25T09:10:26.792 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542175 2024-04-25T09:10:26.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-141733920775 1-60129542175' 2024-04-25T09:10:26.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:26.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:10:26.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=94489280533 2024-04-25T09:10:26.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 94489280533 2024-04-25T09:10:26.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-141733920775 1-60129542175 2-94489280533' 2024-04-25T09:10:26.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:26.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-141733920775 2024-04-25T09:10:26.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:26.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:10:26.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-141733920775 2024-04-25T09:10:26.952 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:26.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920775 2024-04-25T09:10:26.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 141733920775' 2024-04-25T09:10:26.953 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 141733920775 2024-04-25T09:10:26.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:27.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920774 -lt 141733920775 2024-04-25T09:10:27.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:10:28.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:10:28.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:28.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920776 -lt 141733920775 2024-04-25T09:10:28.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:28.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-60129542175 2024-04-25T09:10:28.869 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:28.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:10:28.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-60129542175 2024-04-25T09:10:28.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:28.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542175 2024-04-25T09:10:28.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 60129542175' 2024-04-25T09:10:28.875 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 60129542175 2024-04-25T09:10:28.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:29.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542175 -lt 60129542175 2024-04-25T09:10:29.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:29.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-94489280533 2024-04-25T09:10:29.307 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:29.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:10:29.310 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-94489280533 2024-04-25T09:10:29.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:29.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=94489280533 2024-04-25T09:10:29.312 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 94489280533 2024-04-25T09:10:29.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 94489280533' 2024-04-25T09:10:29.312 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:10:29.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 94489280533 -lt 94489280533 2024-04-25T09:10:29.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:10:29.754 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:29.754 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:30.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:10:30.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:10:30.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:10:30.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:10:30.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:10:30.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:10:30.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:10:30.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:10:30.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:10:30.734 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:10:30.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:30.735 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:31.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:10:31.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:10:31.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:10:31.296 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:10:31.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-04-25T09:10:31.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-04-25T09:10:31.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-04-25T09:10:31.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-04-25T09:10:31.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-04-25T09:10:31.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-04-25T09:10:31.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-04-25T09:10:31.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-04-25T09:10:31.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-04-25T09:10:31.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.637 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-04-25T09:10:31.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-04-25T09:10:31.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-04-25T09:10:31.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-04-25T09:10:31.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-04-25T09:10:31.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.789 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-04-25T09:10:31.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-04-25T09:10:31.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.850 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-04-25T09:10:31.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-04-25T09:10:31.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-04-25T09:10:31.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-04-25T09:10:31.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:31.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:31.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-04-25T09:10:31.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-04-25T09:10:32.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-04-25T09:10:32.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-04-25T09:10:32.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-04-25T09:10:32.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-04-25T09:10:32.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-04-25T09:10:32.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-04-25T09:10:32.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-04-25T09:10:32.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-04-25T09:10:32.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-04-25T09:10:32.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-04-25T09:10:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-04-25T09:10:32.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-04-25T09:10:32.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-04-25T09:10:32.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-04-25T09:10:32.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-04-25T09:10:32.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-04-25T09:10:32.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-04-25T09:10:32.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-04-25T09:10:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-04-25T09:10:32.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-04-25T09:10:32.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-04-25T09:10:32.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-04-25T09:10:32.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-04-25T09:10:32.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-04-25T09:10:32.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-04-25T09:10:32.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-04-25T09:10:32.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-04-25T09:10:32.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-04-25T09:10:32.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.949 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:32.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:32.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-04-25T09:10:32.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-04-25T09:10:33.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-04-25T09:10:33.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-04-25T09:10:33.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-04-25T09:10:33.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-04-25T09:10:33.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-04-25T09:10:33.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-04-25T09:10:33.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-04-25T09:10:33.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-04-25T09:10:33.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-04-25T09:10:33.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-04-25T09:10:33.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-04-25T09:10:33.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-04-25T09:10:33.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-04-25T09:10:33.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-04-25T09:10:33.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-04-25T09:10:33.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-04-25T09:10:33.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-04-25T09:10:33.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-04-25T09:10:33.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-04-25T09:10:33.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-04-25T09:10:33.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-04-25T09:10:33.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.720 INFO:tasks.workunit.client.0.smithi031.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-04-25T09:10:33.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-04-25T09:10:33.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-04-25T09:10:33.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-04-25T09:10:33.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-04-25T09:10:33.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-04-25T09:10:33.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-04-25T09:10:33.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-04-25T09:10:33.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-04-25T09:10:33.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:33.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:33.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-04-25T09:10:33.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-04-25T09:10:34.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-04-25T09:10:34.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-04-25T09:10:34.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-04-25T09:10:34.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-04-25T09:10:34.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-04-25T09:10:34.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-04-25T09:10:34.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-04-25T09:10:34.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-04-25T09:10:34.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-04-25T09:10:34.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-04-25T09:10:34.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-04-25T09:10:34.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-04-25T09:10:34.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-04-25T09:10:34.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:10:34.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-04-25T09:10:34.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:10:34.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-04-25T09:10:34.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:10:34.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:10:35.055 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test-pool' does not exist 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:35.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:35.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:35.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:35.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:35.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:35.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:35.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:35.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:35.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:35.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:35.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:35.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:35.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:35.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:35.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:35.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:10:35.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:35.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.212 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:10:35.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:10:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:10:35.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:35.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:35.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:35.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:35.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:35.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:35.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:35.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:35.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:35.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:35.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:35.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:35.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:10:35.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:35.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.383712 2024-04-25T09:10:35.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.383712 2024-04-25T09:10:35.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:35.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:35.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:10:35.224 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:10:35.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:10:35.275 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-04-25T09:10:35.276 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:10:35.339 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:10:35.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:10:35.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:10:35.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:10:35.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reuse-id 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-04-25T09:10:35.340 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-04-25T09:10:35.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none ' 2024-04-25T09:10:35.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-04-25T09:10:35.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-04-25T09:10:35.341 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:35.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:35.344 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:35.344 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:35.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:35.345 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:35.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:35.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:35.346 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:35.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:35.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:35.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:35.348 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:35.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:35.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:10:35.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:10:35.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:35.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.350 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:35.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.438415 2024-04-25T09:10:35.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:35.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:35.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-04-25T09:10:35.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:10:35.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:35.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.438415 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:10:35.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:10:35.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-04-25T09:10:35.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:10:35.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:10:35.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:35.814 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:35.814 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:35.814 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.815 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:35.815 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:10:35.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:10:36.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:10:36.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:10:36.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:10:36.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:10:36.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:10:36.027 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:10:36.027 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:10:36.027 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:10:36.027 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:10:36.028 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:36.028 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:36.028 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:36.028 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.438415/ceph-mon.a.asok 2024-04-25T09:10:36.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:10:36.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.438415/ceph-mon.a.asok config get fsid 2024-04-25T09:10:36.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:10:36.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:10:36.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:10:36.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:10:36.144 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:10:36.145 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:10:36.145 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:10:36.145 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:10:36.145 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:36.146 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:36.146 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:36.146 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.438415/ceph-mon.a.asok 2024-04-25T09:10:36.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:10:36.146 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.438415/ceph-mon.a.asok config get mon_host 2024-04-25T09:10:36.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-04-25T09:10:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-04-25T09:10:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:10:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:10:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:10:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-04-25T09:10:36.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:10:37.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:10:37.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:37.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:37.140 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:37.141 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:37.141 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:37.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:10:37.141 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:10:37.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-04-25T09:10:37.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:37.175 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:10:37.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:10:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-04-25T09:10:37.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:10:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5e98963-7203-4ffb-abd1-3ccb907e7a7f 2024-04-25T09:10:37.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b5e98963-7203-4ffb-abd1-3ccb907e7a7f' 2024-04-25T09:10:37.178 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 b5e98963-7203-4ffb-abd1-3ccb907e7a7f 2024-04-25T09:10:37.178 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:10:37.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANHipmfYc/DBAA2lUQMj/OLMlmLohOJhmAqg== 2024-04-25T09:10:37.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANHipmfYc/DBAA2lUQMj/OLMlmLohOJhmAqg=="}' 2024-04-25T09:10:37.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5e98963-7203-4ffb-abd1-3ccb907e7a7f -i td/osd-reuse-id/0/new.json 2024-04-25T09:10:37.478 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:10:37.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-04-25T09:10:37.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANHipmfYc/DBAA2lUQMj/OLMlmLohOJhmAqg== --osd-uuid b5e98963-7203-4ffb-abd1-3ccb907e7a7f 2024-04-25T09:10:37.522 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.534+0000 7fb6e02b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:37.547 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.558+0000 7fb6e02b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:37.548 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.558+0000 7fb6e02b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:37.549 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.558+0000 7fb6e02b07c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:10:37.549 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.558+0000 7fb6e02b07c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:10:37.549 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.558+0000 7fb6e02b07c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:10:37.587 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:37.598+0000 7fb6e02b07c0 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-04-25T09:10:43.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-04-25T09:10:43.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:10:43.288 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:10:43.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:10:43.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:10:43.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:10:43.858 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:10:43.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:10:43.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:10:43.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:10:43.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:10:43.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:43.890+0000 7f53e4d527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:43.901 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:43.910+0000 7f53e4d527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:43.902 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:43.914+0000 7f53e4d527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:44.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:10:44.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:10:44.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:10:44.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:10:44.342 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:10:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:10:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:10:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:44.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:44.491 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:44.502+0000 7f53e4d527c0 -1 Falling back to public interface 2024-04-25T09:10:44.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:45.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:10:45.800 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:10:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:45.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:46.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:47.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:10:47.066+0000 7f53e4d527c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:10:47.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:47.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:47.259 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:10:47.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:10:47.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:47.260 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:47.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:48.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:48.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:48.823 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:10:48.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:10:48.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:48.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:49.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:50.255 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:10:50.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:10:50.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:50.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:50.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:51.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:51.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:51.780 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:10:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:10:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:51.781 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:52.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:53.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:53.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:53.223 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:10:53.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:10:53.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:53.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:53.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:54.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:54.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:54.790 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:10:54.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:10:54.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:54.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:55.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:56.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:56.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:56.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:10:56.218 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:10:56.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:56.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:56.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:57.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:57.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:57.643 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:10:57.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:10:57.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:57.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:58.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:59.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:59.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:59.085 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:10:59.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:10:59.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:59.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:59.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:00.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:00.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:00.837 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:11:00.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:11:00.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:00.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:01.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:02.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:02.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:02.318 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:11:02.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:11:02.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:02.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:03.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:04.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:04.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:04.249 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:11:04.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:11:04.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:04.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:05.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:06.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:06.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:06.095 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:11:06.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:11:06.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:06.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:07.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:08.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:08.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:08.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:11:08.005 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:11:08.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:08.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:08.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:09.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:09.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:09.921 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:11:09.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:11:09.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:09.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:10.348 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/249578965,v1:127.0.0.1:6803/249578965] [v2:127.0.0.1:6804/249578965,v1:127.0.0.1:6805/249578965] exists,up b5e98963-7203-4ffb-abd1-3ccb907e7a7f 2024-04-25T09:11:10.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:10.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:10.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:10.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:11:10.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:11:10.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:11:10.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:11:10.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:11:10.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:11:10.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:11:10.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:11:10.354 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:11:10.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1db7d7ec-5fc4-4a06-8d67-0dbd532e283f 2024-04-25T09:11:10.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1db7d7ec-5fc4-4a06-8d67-0dbd532e283f' 2024-04-25T09:11:10.355 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 1db7d7ec-5fc4-4a06-8d67-0dbd532e283f 2024-04-25T09:11:10.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:11:10.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuHipmQe/MFhAAD7pi9DAtdwk/mFNwK0SglQ== 2024-04-25T09:11:10.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuHipmQe/MFhAAD7pi9DAtdwk/mFNwK0SglQ=="}' 2024-04-25T09:11:10.369 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1db7d7ec-5fc4-4a06-8d67-0dbd532e283f -i td/osd-reuse-id/1/new.json 2024-04-25T09:11:10.888 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:11:10.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:11:10.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuHipmQe/MFhAAD7pi9DAtdwk/mFNwK0SglQ== --osd-uuid 1db7d7ec-5fc4-4a06-8d67-0dbd532e283f 2024-04-25T09:11:10.939 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:10.949+0000 7f6f6edbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:10.941 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:10.953+0000 7f6f6edbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:10.943 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:10.953+0000 7f6f6edbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:10.943 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:10.953+0000 7f6f6edbb7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:11:10.943 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:10.953+0000 7f6f6edbb7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:11:10.944 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:10.953+0000 7f6f6edbb7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:11:11.044 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:11.057+0000 7f6f6edbb7c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:11:16.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:11:16.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:11:16.972 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:11:16.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:11:16.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:11:17.539 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:11:17.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:11:17.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:11:17.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:11:17.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:11:17.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:11:17.570 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:17.581+0000 7f9e2aff47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:17.581 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:17.589+0000 7f9e2aff47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:17.582 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:17.593+0000 7f9e2aff47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:18.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:11:18.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:11:18.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:11:18.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:11:18.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:11:18.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:18.010 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:11:18.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:11:18.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:18.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:18.154 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:18.165+0000 7f9e2aff47c0 -1 Falling back to public interface 2024-04-25T09:11:18.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:19.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:19.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:19.441 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:11:19.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:11:19.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:19.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:19.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:20.779 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:20.789+0000 7f9e2aff47c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:11:20.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:20.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:20.859 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:11:20.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:11:20.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:20.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:21.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:22.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:22.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:22.428 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:11:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:11:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:22.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:22.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:23.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:23.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:23.865 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:11:23.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:11:23.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:23.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:24.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:25.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:25.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:25.321 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:11:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:11:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:25.846 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:26.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:26.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:26.848 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:11:26.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:11:26.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:26.849 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:27.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:28.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:11:28.391 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:11:28.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:28.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:28.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:29.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:29.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:29.869 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:11:29.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:11:29.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:29.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:30.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:31.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:31.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:31.457 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:11:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:11:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:31.901 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:32.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:32.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:32.903 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:11:32.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:11:32.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:32.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:33.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:34.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:34.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:34.405 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:11:34.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:11:34.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:34.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:35.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:36.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:36.372 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:36.372 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:11:36.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:11:36.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:36.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:37.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:38.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:38.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:38.062 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:11:38.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:11:38.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:38.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:39.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:40.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:40.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:40.232 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:11:40.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:11:40.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:40.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:40.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:41.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:41.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:41.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:11:41.719 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:11:41.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:41.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:42.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:43.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:43.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:43.167 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:11:43.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:11:43.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:43.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:43.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:44.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:44.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:44.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:11:44.611 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T09:11:44.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:44.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:45.087 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/248622127,v1:127.0.0.1:6811/248622127] [v2:127.0.0.1:6812/248622127,v1:127.0.0.1:6813/248622127] exists,up 1db7d7ec-5fc4-4a06-8d67-0dbd532e283f 2024-04-25T09:11:45.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:45.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:45.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:45.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-04-25T09:11:45.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T09:11:45.561 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' does not exist 2024-04-25T09:11:45.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T09:11:45.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T09:11:46.121 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'rbd' already exists 2024-04-25T09:11:46.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:11:47.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T09:11:48.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-04-25T09:11:48.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:11:48.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:11:48.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:11:48.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:11:48.382 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:11:48.382 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:11:48.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:11:48.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:11:48.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:11:48.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:11:48.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:11:48.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:11:48.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:11:48.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:11:48.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:11:48.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:11:48.977 INFO:tasks.workunit.client.0.smithi031.stderr:1' 2024-04-25T09:11:48.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:11:48.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:11:48.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:11:49.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-04-25T09:11:49.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-04-25T09:11:49.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-04-25T09:11:49.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:11:49.151 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:11:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-04-25T09:11:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-04-25T09:11:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2024-04-25T09:11:49.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:11:49.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-04-25T09:11:49.295 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:11:49.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:11:49.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-04-25T09:11:49.297 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:11:49.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-04-25T09:11:49.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-04-25T09:11:49.298 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836492 2024-04-25T09:11:49.298 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:49.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836492 2024-04-25T09:11:49.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:11:50.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:11:50.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:51.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836492 2024-04-25T09:11:51.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:11:51.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-04-25T09:11:51.182 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:11:51.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:11:51.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-04-25T09:11:51.184 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:11:51.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-04-25T09:11:51.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-25T09:11:51.186 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672969 2024-04-25T09:11:51.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:11:51.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-25T09:11:51.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:11:51.602 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:51.602 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:52.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-25T09:11:52.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:11:52.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:11:52.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:11:52.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:11:52.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:11:52.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:11:52.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:11:52.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-25T09:11:52.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:11:52.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:52.580 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:53.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-25T09:11:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:11:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:11:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-04-25T09:11:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-04-25T09:11:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-04-25T09:11:53.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-04-25T09:11:53.750 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already out. 2024-04-25T09:11:53.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-04-25T09:11:53.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:53.770 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:53.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:53.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:53.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:54.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:54.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-04-25T09:11:54.572 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 is already down. 2024-04-25T09:11:54.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-04-25T09:11:55.095 INFO:tasks.workunit.client.0.smithi031.stderr:osd.1 does not exist 2024-04-25T09:11:55.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-04-25T09:11:55.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-04-25T09:11:55.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:11:55.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-04-25T09:11:55.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:55.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:55.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:55.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:55.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:55.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:11:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:11:55.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:11:55.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:11:55.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:11:55.121 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:11:55.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:55.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:11:55.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:11:55.122 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:55.123 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:11:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:11:55.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-04-25T09:11:55.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:11:55.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:55.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:11:55.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.438415 2024-04-25T09:11:55.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:11:55.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:11:55.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-04-25T09:11:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:11:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:11:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:11:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:11:55.134 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:55.135 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:11:55.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:11:55.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:11:55.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:11:55.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:11:55.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:11:55.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab8d3bf3-12bb-4df2-8cfc-dd2baf3b790a 2024-04-25T09:11:55.137 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 ab8d3bf3-12bb-4df2-8cfc-dd2baf3b790a 2024-04-25T09:11:55.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ab8d3bf3-12bb-4df2-8cfc-dd2baf3b790a' 2024-04-25T09:11:55.138 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:11:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbHipm2LJJChAASRdu1yG66AMTb74Q5qldGg== 2024-04-25T09:11:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbHipm2LJJChAASRdu1yG66AMTb74Q5qldGg=="}' 2024-04-25T09:11:55.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab8d3bf3-12bb-4df2-8cfc-dd2baf3b790a -i td/osd-reuse-id/1/new.json 2024-04-25T09:11:55.663 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:11:55.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:11:55.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbHipm2LJJChAASRdu1yG66AMTb74Q5qldGg== --osd-uuid ab8d3bf3-12bb-4df2-8cfc-dd2baf3b790a 2024-04-25T09:11:55.713 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:55.715 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:55.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:55.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 asok(0x55873d1a4000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.438415/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:11:55.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:11:55.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:11:55.717 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.725+0000 7fa4c82797c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:11:55.748 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:11:55.757+0000 7fa4c82797c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:12:01.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:12:01.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:12:01.624 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:12:01.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:12:01.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:12:02.197 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:12:02.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:12:02.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e25410f3-878c-493d-ba43-6ca82cc17637 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.438415/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:12:02.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:12:02.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:12:02.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:12:02.227 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:02.236+0000 7fed19c0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:02.236 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:02.244+0000 7fed19c0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:02.237 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:02.244+0000 7fed19c0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:02.237 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:02.248+0000 7fed19c0d7c0 -1 asok(0x559a3944a000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.438415/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:12:02.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:12:02.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:12:02.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:02.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:02.810 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:02.820+0000 7fed19c0d7c0 -1 Falling back to public interface 2024-04-25T09:12:03.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:04.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:04.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:04.144 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:12:04.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:12:04.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:04.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:04.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:05.327 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:05.336+0000 7fed19c0d7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:12:05.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:05.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:05.628 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:12:05.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:12:05.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:05.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:06.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:07.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:07.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:07.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:12:07.075 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:12:07.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:07.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:07.545 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2783470021,v1:127.0.0.1:6811/2783470021] [v2:127.0.0.1:6812/2783470021,v1:127.0.0.1:6813/2783470021] exists,up ab8d3bf3-12bb-4df2-8cfc-dd2baf3b790a 2024-04-25T09:12:07.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:12:07.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:12:07.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:12:07.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-04-25T09:12:07.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:12:07.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:12:07.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:12:07.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:12:07.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:12:07.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:12:07.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:12:07.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:12:07.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:12:07.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:12:07.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:12:07.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:12:07.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:12:07.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:12:07.658 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:12:07.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:12:07.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:12:07.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:12:07.659 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:12:07.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:12:07.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:12:07.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:12:07.671 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:12:07.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:07.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:12:07.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.438415 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reuse-id 0 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:12:07.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:12:07.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:12:07.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:12:07.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:12:07.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:12:07.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:12:07.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:12:07.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:12:07.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:12:07.679 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:12:07.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:12:07.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:12:07.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:12:07.681 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:12:07.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:12:07.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:12:07.682 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:12:07.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:12:07.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:12:07.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:12:07.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:12:07.685 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:07.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.438415 2024-04-25T09:12:07.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.438415 2024-04-25T09:12:07.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:12:07.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:12:07.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:12:07.688 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:12:07.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:12:07.740 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-04-25T09:12:07.740 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-04-25T09:12:07.792 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:12:07.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/pg-split-merge 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-04-25T09:12:07.798 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-04-25T09:12:07.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none ' 2024-04-25T09:12:07.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:12:07.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-04-25T09:12:07.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_import_after_merge_and_gap 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_import_after_split' 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:12:07.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:12:07.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:12:07.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:12:07.803 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:12:07.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:12:07.804 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:12:07.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:12:07.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:12:07.806 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:12:07.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:12:07.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:12:07.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:12:07.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:12:07.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:12:07.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:12:07.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:12:07.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:12:07.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:07.810 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:07.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:12:07.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:12:07.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:12:07.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:12:07.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:12:07.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:07.812 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:07.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.443986 2024-04-25T09:12:07.813 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:12:07.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-04-25T09:12:08.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:12:08.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:08.293 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:08.294 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:08.294 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:08.294 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:08.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:12:08.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T09:12:08.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:12:08.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:12:08.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:12:08.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:12:08.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:12:08.507 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:12:08.508 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:12:08.508 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:12:08.508 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:12:08.508 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:08.508 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:08.509 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:08.509 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.443986/ceph-mon.a.asok 2024-04-25T09:12:08.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:12:08.509 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.443986/ceph-mon.a.asok config get fsid 2024-04-25T09:12:08.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:12:08.627 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:12:08.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:12:08.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:12:08.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:12:08.628 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:12:08.628 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:12:08.628 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:12:08.629 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:08.629 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:08.629 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:08.629 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.443986/ceph-mon.a.asok 2024-04-25T09:12:08.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:12:08.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.443986/ceph-mon.a.asok config get mon_host 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:12:08.741 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:12:09.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:12:09.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:12:09.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:09.738 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:12:09.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:12:09.740 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:12:09.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bdfb3206-89a8-4a4b-9a7c-78eda0ba3993 2024-04-25T09:12:09.740 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 bdfb3206-89a8-4a4b-9a7c-78eda0ba3993 2024-04-25T09:12:09.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bdfb3206-89a8-4a4b-9a7c-78eda0ba3993' 2024-04-25T09:12:09.741 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:12:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpHipmIxUMLhAAre/TGv+cccgYWmZCMr28rw== 2024-04-25T09:12:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpHipmIxUMLhAAre/TGv+cccgYWmZCMr28rw=="}' 2024-04-25T09:12:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bdfb3206-89a8-4a4b-9a7c-78eda0ba3993 -i td/pg-split-merge/0/new.json 2024-04-25T09:12:10.042 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:12:10.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:12:10.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpHipmIxUMLhAAre/TGv+cccgYWmZCMr28rw== --osd-uuid bdfb3206-89a8-4a4b-9a7c-78eda0ba3993 2024-04-25T09:12:10.093 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.104+0000 7fc973ca97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:10.101 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.112+0000 7fc973ca97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:10.102 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.112+0000 7fc973ca97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:10.103 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.112+0000 7fc973ca97c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:12:10.103 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.112+0000 7fc973ca97c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:12:10.103 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.112+0000 7fc973ca97c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:12:10.141 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:10.152+0000 7fc973ca97c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:12:16.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:12:16.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:12:16.380 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:12:16.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:12:16.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:12:16.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:12:16.981 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:12:16.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:12:16.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:12:16.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:12:16.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:12:17.017 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:17.024+0000 7f86bd8c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:17.061 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:17.072+0000 7f86bd8c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:17.063 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:17.072+0000 7f86bd8c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:12:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:12:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:12:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:12:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:12:17.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:17.413 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:12:17.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:12:17.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:17.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:17.603 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:17.612+0000 7f86bd8c37c0 -1 Falling back to public interface 2024-04-25T09:12:17.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:18.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:18.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:18.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:12:18.911 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:12:18.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:18.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:19.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:20.240 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:20.248+0000 7f86bd8c37c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:12:20.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:20.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:20.393 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:12:20.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:12:20.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:20.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:20.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:21.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:21.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:21.899 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:12:21.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:12:21.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:21.899 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:22.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:23.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:23.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:23.377 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:12:23.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:12:23.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:23.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:23.817 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:24.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:24.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:24.820 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:12:24.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:12:24.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:24.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:25.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:26.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:26.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:26.400 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:12:26.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:12:26.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:26.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:26.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:27.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:27.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:27.864 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:12:27.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:12:27.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:27.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:28.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:29.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:29.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:29.362 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:12:29.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:12:29.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:29.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:29.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:30.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:30.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:30.798 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:12:30.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:12:30.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:30.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:31.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:32.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:32.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:32.313 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:12:32.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:12:32.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:32.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:33.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:34.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:34.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:34.085 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:12:34.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:12:34.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:34.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:35.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:36.042 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:36.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:36.043 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:12:36.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:12:36.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:36.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:36.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:37.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:37.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:37.873 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:12:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:12:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:38.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:39.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:39.640 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:39.640 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:12:39.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:12:39.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:39.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:40.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:41.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:41.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:41.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:12:41.367 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:12:41.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:41.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:42.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:43.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:43.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:43.069 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:12:43.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:12:43.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:43.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:43.518 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/274478713,v1:127.0.0.1:6803/274478713] [v2:127.0.0.1:6804/274478713,v1:127.0.0.1:6805/274478713] exists,up bdfb3206-89a8-4a4b-9a7c-78eda0ba3993 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:12:43.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:12:43.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:12:43.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-04-25T09:12:43.525 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:12:43.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75ba2462-5c3d-4756-a22e-3eda4cae35ca 2024-04-25T09:12:43.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 75ba2462-5c3d-4756-a22e-3eda4cae35ca' 2024-04-25T09:12:43.526 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 75ba2462-5c3d-4756-a22e-3eda4cae35ca 2024-04-25T09:12:43.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:12:43.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLHipmcUcgIRAAkq2qAuulo0zrH79LHE8MqQ== 2024-04-25T09:12:43.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLHipmcUcgIRAAkq2qAuulo0zrH79LHE8MqQ=="}' 2024-04-25T09:12:43.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75ba2462-5c3d-4756-a22e-3eda4cae35ca -i td/pg-split-merge/1/new.json 2024-04-25T09:12:44.042 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:12:44.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-04-25T09:12:44.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLHipmcUcgIRAAkq2qAuulo0zrH79LHE8MqQ== --osd-uuid 75ba2462-5c3d-4756-a22e-3eda4cae35ca 2024-04-25T09:12:44.096 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.108+0000 7f26898357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:44.099 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.112+0000 7f26898357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:44.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.112+0000 7f26898357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:44.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.112+0000 7f26898357c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:12:44.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.112+0000 7f26898357c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:12:44.100 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.112+0000 7f26898357c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:12:44.144 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:44.156+0000 7f26898357c0 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-04-25T09:12:50.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-04-25T09:12:50.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:12:50.424 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:12:50.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:12:50.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:12:51.014 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:12:51.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:12:51.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:12:51.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:12:51.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:12:51.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:12:51.049 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:51.060+0000 7f0f26fb77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:51.052 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:51.064+0000 7f0f26fb77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:51.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:51.068+0000 7f0f26fb77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:51.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:12:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:12:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:12:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:12:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:12:51.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:51.499 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:12:51.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:12:51.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:51.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:51.650 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:51.664+0000 7f0f26fb77c0 -1 Falling back to public interface 2024-04-25T09:12:51.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:52.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:52.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:52.967 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:12:52.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:12:52.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:52.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:53.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:54.220 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:12:54.232+0000 7f0f26fb77c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:12:54.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:54.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:54.406 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:12:54.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:12:54.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:54.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:54.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:55.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:55.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:55.847 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:12:55.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:12:55.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:55.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:56.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:57.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:57.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:57.397 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:12:57.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:12:57.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:57.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:57.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:58.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:58.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:58.829 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:12:58.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:12:58.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:58.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:59.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:00.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:00.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:00.452 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:13:00.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:13:00.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:00.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:00.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:01.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:01.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:01.971 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:13:01.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:13:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:01.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:02.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:03.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:03.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:03.430 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:13:03.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:13:03.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:03.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:03.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:04.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:04.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:04.973 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:13:04.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:13:04.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:04.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:05.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:06.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:06.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:06.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:13:06.524 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:13:06.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:06.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:07.615 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:08.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:08.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:08.617 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:13:08.617 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:13:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:08.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:09.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:10.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:10.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:13:10.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:13:10.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:10.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:11.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:12.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:12.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:12.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:13:12.662 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:13:12.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:12.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:13.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:14.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:14.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:14.600 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:13:14.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:13:14.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:14.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:15.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:16.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:16.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:16.707 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:13:16.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:13:16.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:16.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:17.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:18.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:18.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:18.150 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:13:18.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:13:18.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:18.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:18.640 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1280035339,v1:127.0.0.1:6811/1280035339] [v2:127.0.0.1:6812/1280035339,v1:127.0.0.1:6813/1280035339] exists,up 75ba2462-5c3d-4756-a22e-3eda4cae35ca 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:13:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:13:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:13:18.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:13:18.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=840b4b65-4df3-43cd-82e1-649c074428d8 2024-04-25T09:13:18.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 840b4b65-4df3-43cd-82e1-649c074428d8' 2024-04-25T09:13:18.648 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 840b4b65-4df3-43cd-82e1-649c074428d8 2024-04-25T09:13:18.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:13:18.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuHipmSpXbKBAARuT+tKf1iyCfNI/9Ap4yJw== 2024-04-25T09:13:18.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuHipmSpXbKBAARuT+tKf1iyCfNI/9Ap4yJw=="}' 2024-04-25T09:13:18.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 840b4b65-4df3-43cd-82e1-649c074428d8 -i td/pg-split-merge/2/new.json 2024-04-25T09:13:19.166 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:13:19.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-04-25T09:13:19.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCuHipmSpXbKBAARuT+tKf1iyCfNI/9Ap4yJw== --osd-uuid 840b4b65-4df3-43cd-82e1-649c074428d8 2024-04-25T09:13:19.222 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.235+0000 7f831af047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:19.230 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.243+0000 7f831af047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:19.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.243+0000 7f831af047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:19.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.243+0000 7f831af047c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:13:19.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.243+0000 7f831af047c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:13:19.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.243+0000 7f831af047c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:13:19.270 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:19.283+0000 7f831af047c0 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-04-25T09:13:25.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-04-25T09:13:25.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:13:25.428 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:13:25.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:13:25.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:13:26.004 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:13:26.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:13:26.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:13:26.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:13:26.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:13:26.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:13:26.039 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:26.051+0000 7fc6036047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:26.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:26.067+0000 7fc6036047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:26.055 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:26.067+0000 7fc6036047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:26.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:13:26.470 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:13:26.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:26.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:26.624 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:26.635+0000 7fc6036047c0 -1 Falling back to public interface 2024-04-25T09:13:26.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:27.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:27.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:27.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:13:27.953 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:13:27.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:27.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:28.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:29.240 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:13:29.251+0000 7fc6036047c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:13:29.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:29.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:29.418 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:13:29.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:13:29.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:29.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:29.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:30.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:30.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:30.930 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:13:30.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:13:30.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:30.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:31.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:32.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:32.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:32.370 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:13:32.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:13:32.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:32.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:32.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:33.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:33.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:33.830 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:13:33.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:13:33.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:33.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:34.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:35.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:35.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:35.365 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:13:35.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:13:35.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:35.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:35.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:36.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:36.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:36.806 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:13:36.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:13:36.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:36.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:37.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:38.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:38.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:38.352 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:13:38.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:13:38.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:38.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:38.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:39.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:39.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:39.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:13:39.842 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:13:39.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:39.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:40.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:41.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:41.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:41.401 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:13:41.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:13:41.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:41.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:42.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:43.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:43.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:43.217 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:13:43.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:13:43.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:43.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:43.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:44.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:44.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:44.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:13:44.956 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:13:44.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:44.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:45.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:46.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:46.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:46.665 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:13:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:13:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:46.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:47.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:48.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:13:48.117 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:13:48.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:48.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:48.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:49.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:49.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:49.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:13:49.842 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:13:49.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:49.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:51.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:52.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:52.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:52.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:13:52.284 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:13:52.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:52.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:52.728 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1940671744,v1:127.0.0.1:6819/1940671744] [v2:127.0.0.1:6820/1940671744,v1:127.0.0.1:6821/1940671744] exists,up 840b4b65-4df3-43cd-82e1-649c074428d8 2024-04-25T09:13:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:13:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:13:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:13:52.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-04-25T09:13:53.274 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T09:13:53.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-04-25T09:13:54.720 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 pgp_num to 1 2024-04-25T09:13:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:13:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:13:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:13:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:13:54.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:13:54.747 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:13:54.747 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:13:54.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:13:54.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:13:54.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:13:54.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:13:54.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:13:54.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:13:54.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:13:54.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:13:54.864 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:13:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:13:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:13:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:13:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:13:55.417 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:13:55.418 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:13:55.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T09:13:55.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T09:13:55.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T09:13:55.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:13:55.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:13:55.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672972 2024-04-25T09:13:55.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672972 2024-04-25T09:13:55.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672972' 2024-04-25T09:13:55.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:13:55.752 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:13:55.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509446 2024-04-25T09:13:55.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509446 2024-04-25T09:13:55.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672972 2-64424509446' 2024-04-25T09:13:55.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:13:55.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T09:13:55.906 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:13:55.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:13:55.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T09:13:55.907 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:13:55.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T09:13:55.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T09:13:55.909 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836500 2024-04-25T09:13:55.909 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:13:56.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836500 2024-04-25T09:13:56.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:13:57.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:13:57.349 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:13:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836500 2024-04-25T09:13:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:13:57.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672972 2024-04-25T09:13:57.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:13:57.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:13:57.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672972 2024-04-25T09:13:57.958 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:13:57.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672972 2024-04-25T09:13:57.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-04-25T09:13:57.960 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672972 2024-04-25T09:13:57.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:13:58.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672972 -lt 42949672972 2024-04-25T09:13:58.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:13:58.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509446 2024-04-25T09:13:58.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:13:58.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:13:58.433 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509446 2024-04-25T09:13:58.434 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:13:58.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509446 2024-04-25T09:13:58.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-25T09:13:58.435 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509446 2024-04-25T09:13:58.435 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:13:58.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509446 2024-04-25T09:13:58.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:13:58.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:58.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:59.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:13:59.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:59.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:59.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:59.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:59.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:59.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:59.464 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:59.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:13:59.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:59.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:59.875 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:00.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:14:00.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:14:00.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:14:00.418 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-04-25T09:14:00.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-04-25T09:14:00.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-04-25T09:14:00.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-04-25T09:14:01.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-04-25T09:14:01.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-04-25T09:14:01.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-04-25T09:14:01.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-04-25T09:14:01.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-04-25T09:14:01.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-04-25T09:14:02.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-04-25T09:14:02.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:14:02.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:14:02.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:14:02.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:14:02.167 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:14:02.168 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:14:02.168 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:14:02.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:14:02.168 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:14:02.169 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:14:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:14:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:14:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:14:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:14:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:14:02.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:14:02.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:14:02.720 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:14:02.720 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:14:02.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:14:02.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:02.721 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:14:02.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T09:14:02.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T09:14:02.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T09:14:02.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:02.883 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:14:03.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-04-25T09:14:03.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-04-25T09:14:03.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-04-25T09:14:03.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:03.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:14:03.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-04-25T09:14:03.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-04-25T09:14:03.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509450' 2024-04-25T09:14:03.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:03.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T09:14:03.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:03.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:14:03.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T09:14:03.200 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:03.202 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836504 2024-04-25T09:14:03.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T09:14:03.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T09:14:03.202 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:03.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836504 2024-04-25T09:14:03.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:14:04.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:14:04.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:05.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-25T09:14:05.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:05.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-04-25T09:14:05.137 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:05.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:14:05.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-04-25T09:14:05.139 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:05.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-04-25T09:14:05.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-04-25T09:14:05.140 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672976 2024-04-25T09:14:05.140 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:14:05.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-04-25T09:14:05.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:05.566 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-04-25T09:14:05.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:05.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:14:05.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-04-25T09:14:05.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:05.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-04-25T09:14:05.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-25T09:14:05.571 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509450 2024-04-25T09:14:05.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:14:05.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-25T09:14:05.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:14:05.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:05.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:06.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:14:06.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:06.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:06.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:06.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:06.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:06.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:06.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:07.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:14:07.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:07.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:07.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:07.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:14:07.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:14:07.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:14:07.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-04-25T09:14:07.705 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T09:14:07.705 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "" 2024-04-25T09:14:07.705 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T09:14:07.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-04-25T09:14:08.552 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 2 2024-04-25T09:14:08.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:14:08.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:14:08.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:14:08.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:14:08.577 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:14:08.577 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:14:08.577 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:14:08.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:14:08.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:14:08.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:14:08.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:14:08.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:14:08.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:14:08.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:14:08.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:14:08.699 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:14:09.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:14:09.333 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:14:09.334 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:14:09.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:14:09.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:09.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:14:09.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-04-25T09:14:09.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-04-25T09:14:09.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-04-25T09:14:09.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:09.518 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:14:09.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-04-25T09:14:09.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-04-25T09:14:09.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672979' 2024-04-25T09:14:09.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:09.683 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:14:09.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672979 2-64424509453' 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:09.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:14:09.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-04-25T09:14:09.843 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:09.844 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-04-25T09:14:09.844 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836507 2024-04-25T09:14:09.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-04-25T09:14:09.845 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:10.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836507 2024-04-25T09:14:10.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:14:11.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:14:11.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:11.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-04-25T09:14:11.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:11.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-04-25T09:14:11.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:11.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:14:11.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-04-25T09:14:11.750 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:11.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-04-25T09:14:11.752 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672979 2024-04-25T09:14:11.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-25T09:14:11.753 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:14:12.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672979 2024-04-25T09:14:12.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:12.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-04-25T09:14:12.213 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:12.214 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:14:12.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-04-25T09:14:12.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:12.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-04-25T09:14:12.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-25T09:14:12.217 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509453 2024-04-25T09:14:12.217 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:14:12.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509453 2024-04-25T09:14:12.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:14:12.706 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:12.707 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:13.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:14:13.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:13.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:13.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:13.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:13.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:13.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:13.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:13.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:14:13.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:13.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:13.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:14:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:14:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:14:14.246 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-04-25T09:14:14.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:14:14.247 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:14:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:14:14.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:14:14.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:14:14.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:14:14.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-04-25T09:14:15.713 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:14:15.714 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:14:15.714 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:14:15.787 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:14:16.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:14:16.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:14:16.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:14:16.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:14:16.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:14:16.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:14:16.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:14:16.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:14:16.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:14:16.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:16.264 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:16.264 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:16.264 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:16.265 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:16.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:14:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:14:16.268 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:14:16.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:14:16.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:16.318+0000 7f86d94d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:16.310 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:16.322+0000 7f86d94d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:16.311 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:16.322+0000 7f86d94d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:16.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:14:16.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:14:16.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:14:16.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:14:16.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:14:16.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:16.706 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:14:16.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:14:16.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:14:16.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:16.859 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:16.870+0000 7f86d94d67c0 -1 Falling back to public interface 2024-04-25T09:14:17.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:18.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:18.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:18.205 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:14:18.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:14:18.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:18.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:14:18.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:19.597 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:19.610+0000 7f86d94d67c0 -1 osd.2 26 log_to_monitors true 2024-04-25T09:14:19.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:19.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:19.632 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:14:19.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:14:19.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:19.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:14:20.174 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 29 up_thru 17 down_at 27 last_clean_interval [15,26) [v2:127.0.0.1:6818/4237230597,v1:127.0.0.1:6819/4237230597] [v2:127.0.0.1:6820/4237230597,v1:127.0.0.1:6821/4237230597] exists,up 840b4b65-4df3-43cd-82e1-649c074428d8 2024-04-25T09:14:20.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:14:20.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:14:20.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:14:20.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:14:20.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:14:20.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:14:20.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:14:20.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:14:20.177 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:14:20.177 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:14:20.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:14:20.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:14:20.177 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:14:20.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:14:20.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:14:20.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:14:20.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:14:20.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:14:20.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:14:20.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:14:20.715 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:14:20.715 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:14:20.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:14:20.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:20.716 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:14:20.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T09:14:20.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T09:14:20.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T09:14:20.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:20.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:14:21.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T09:14:21.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T09:14:21.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672983' 2024-04-25T09:14:21.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:21.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:14:21.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051587 2024-04-25T09:14:21.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051587 2024-04-25T09:14:21.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672983 2-124554051587' 2024-04-25T09:14:21.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:21.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T09:14:21.180 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:21.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:14:21.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T09:14:21.181 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:21.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T09:14:21.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T09:14:21.183 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836511 2024-04-25T09:14:21.183 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:21.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T09:14:21.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:14:22.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:14:22.597 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:23.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T09:14:23.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:23.039 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T09:14:23.040 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:23.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:14:23.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T09:14:23.042 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:23.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T09:14:23.043 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T09:14:23.043 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672983 2024-04-25T09:14:23.044 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:14:23.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T09:14:23.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:23.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-124554051587 2024-04-25T09:14:23.517 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:23.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:14:23.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-124554051587 2024-04-25T09:14:23.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:23.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051587 2024-04-25T09:14:23.520 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 124554051587 2024-04-25T09:14:23.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 124554051587' 2024-04-25T09:14:23.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:14:23.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051587 -lt 124554051587 2024-04-25T09:14:23.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:14:23.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:23.963 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:24.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:14:24.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:24.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:24.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:24.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:24.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:24.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:24.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:24.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:14:24.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:24.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:24.999 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:25.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:14:25.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:14:25.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:14:25.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-04-25T09:14:25.693 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T09:14:25.693 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "" 2024-04-25T09:14:25.693 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T09:14:25.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-04-25T09:14:26.566 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already out. osd.1 is already out. 2024-04-25T09:14:26.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-04-25T09:14:27.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-04-25T09:14:32.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-04-25T09:14:32.478 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-04-25T09:14:32.478 INFO:tasks.workunit.client.0.smithi031.stdout: "success": "" 2024-04-25T09:14:32.478 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-04-25T09:14:32.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-04-25T09:14:33.035 INFO:tasks.workunit.client.0.smithi031.stderr:osd.0 is already in. osd.1 is already in. 2024-04-25T09:14:33.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-04-25T09:14:34.250 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 3 2024-04-25T09:14:34.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:14:34.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:14:34.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:14:34.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:14:34.278 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:14:34.278 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:14:34.278 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:14:34.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:14:34.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:14:34.279 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:14:34.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:14:34.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:14:34.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:14:34.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:14:34.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:14:34.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:14:34.975 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:14:34.976 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:14:34.976 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:14:34.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:14:34.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:34.976 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:14:35.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-04-25T09:14:35.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-04-25T09:14:35.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-04-25T09:14:35.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:35.143 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:14:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672988 2024-04-25T09:14:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672988 2024-04-25T09:14:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672988' 2024-04-25T09:14:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:35.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:14:35.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051592 2024-04-25T09:14:35.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051592 2024-04-25T09:14:35.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672988 2-124554051592' 2024-04-25T09:14:35.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:35.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-04-25T09:14:35.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:35.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:14:35.551 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-04-25T09:14:35.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:35.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-04-25T09:14:35.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-25T09:14:35.553 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836516 2024-04-25T09:14:35.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:36.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836516 2024-04-25T09:14:36.079 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:14:37.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:14:37.081 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:37.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-04-25T09:14:37.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:37.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672988 2024-04-25T09:14:37.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:37.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:14:37.567 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672988 2024-04-25T09:14:37.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:37.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672988 2024-04-25T09:14:37.570 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672988 2024-04-25T09:14:37.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-04-25T09:14:37.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:14:38.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672988 -lt 42949672988 2024-04-25T09:14:38.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:38.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-124554051592 2024-04-25T09:14:38.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:38.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:14:38.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-124554051592 2024-04-25T09:14:38.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:38.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051592 2024-04-25T09:14:38.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 124554051592' 2024-04-25T09:14:38.032 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 124554051592 2024-04-25T09:14:38.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:14:38.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051592 -lt 124554051592 2024-04-25T09:14:38.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:14:38.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:38.543 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:39.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:14:39.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:39.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:39.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:39.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:39.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:39.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:39.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:39.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:14:39.531 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:39.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:39.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:40.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:14:40.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:14:40.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:14:40.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-04-25T09:14:40.484 INFO:tasks.workunit.client.0.smithi031.stderr:instructing pg 1.0 on osd.1 to scrub 2024-04-25T09:14:40.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-04-25T09:14:50.501 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-04-25T09:14:51.089 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:33.546345+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e37: 3 total, 3 up, 3 in 2024-04-25T09:14:51.089 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:33.906980+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2024-04-25T09:14:51.089 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:34.325738+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2024-04-25T09:14:51.089 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:34.576727+0000 mgr.x (mgr.4100) 70 : cluster [DBG] pgmap v100: 1 pgs: 1 unknown; 82 KiB data, 27 MiB used, 100 GiB / 100 GiB avail 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:34.770694+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:35.020770+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:35.397441+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:35.479596+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (24'7,24'7] MIN to 24'7 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:36.278299+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:36.577213+0000 mgr.x (mgr.4100) 71 : cluster [DBG] pgmap v105: 1 pgs: 1 remapped+peering; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 21 KiB/s wr, 1 op/s 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:36.672876+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e44: 3 total, 3 up, 3 in 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:38.577711+0000 mgr.x (mgr.4100) 72 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 13 KiB/s wr, 1 op/s; 12 KiB/s, 1 objects/s recovering 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:40.578346+0000 mgr.x (mgr.4100) 77 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 8.6 KiB/s wr, 0 op/s; 8.5 KiB/s, 0 objects/s recovering 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:41.152031+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:41.157277+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-04-25T09:14:51.090 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:42.579044+0000 mgr.x (mgr.4100) 78 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 6.6 KiB/s, 0 objects/s recovering 2024-04-25T09:14:51.091 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:44.579726+0000 mgr.x (mgr.4100) 79 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 5.7 KiB/s, 0 objects/s recovering 2024-04-25T09:14:51.091 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:46.580389+0000 mgr.x (mgr.4100) 80 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.7 KiB/s, 0 objects/s recovering 2024-04-25T09:14:51.091 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:48.581094+0000 mgr.x (mgr.4100) 81 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.0 KiB/s, 0 objects/s recovering 2024-04-25T09:14:51.091 INFO:tasks.workunit.client.0.smithi031.stdout:2024-04-25T09:14:50.581775+0000 mgr.x (mgr.4100) 82 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail 2024-04-25T09:14:51.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:14:51.512 INFO:tasks.workunit.client.0.smithi031.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-04-25T09:14:51.513 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 24'7 44:96 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:14:41.157287+0000 2024-04-25T09:13:53.117416+0000 1 periodic scrub scheduled @ 2024-04-26T19:32:36.084995+0000 2024-04-25T09:14:51.513 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T09:14:51.513 INFO:tasks.workunit.client.0.smithi031.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:14:51.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:14:51.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-04-25T09:14:51.957 INFO:tasks.workunit.client.0.smithi031.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 24'7 44:96 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:14:41.157287+0000 2024-04-25T09:13:53.117416+0000 1 periodic scrub scheduled @ 2024-04-26T19:32:36.084995+0000 2024-04-25T09:14:51.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:14:51.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:14:51.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:14:51.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:14:51.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:14:51.959 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:14:51.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:14:51.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:14:51.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:14:52.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:14:52.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:14:52.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:14:52.069 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:14:52.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:14:52.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:14:52.070 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:14:52.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:52.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:14:52.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:14:52.072 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:52.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:14:52.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:14:52.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:14:52.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:14:52.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:52.090 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:52.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:14:52.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:14:52.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:14:52.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:14:52.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:14:52.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:14:52.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:14:52.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:14:52.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:14:52.094 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:14:52.094 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:14:52.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:14:52.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:14:52.095 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:14:52.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:52.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:14:52.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:14:52.096 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:52.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:14:52.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:14:52.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:14:52.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:14:52.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:52.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:52.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.443986 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-04-25T09:14:52.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:14:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:14:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:14:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:14:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:14:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:14:52.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:14:52.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:14:52.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:52.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:52.556 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:52.556 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:52.557 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:52.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:14:52.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:14:52.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:14:52.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:14:52.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:14:52.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:14:52.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:14:52.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.443986/ceph-mon.a.asok 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:14:52.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.443986/ceph-mon.a.asok config get fsid 2024-04-25T09:14:52.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:14:52.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:14:52.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:14:52.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:14:52.890 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:14:52.890 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.443986/ceph-mon.a.asok 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:14:52.891 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.443986/ceph-mon.a.asok config get mon_host 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:14:53.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:14:53.925 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:14:53.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:53.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:53.926 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:53.926 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:53.926 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:53.926 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:14:53.927 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:14:53.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:14:53.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:14:53.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:14:53.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:14:53.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:14:53.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:14:53.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d91c4f80-3723-41df-b7cb-c0ba7c928b9a' 2024-04-25T09:14:53.968 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:14:53.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:14:53.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANHypmqTeYOxAAXa/7uyryLidHgrka09q0yw== 2024-04-25T09:14:53.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANHypmqTeYOxAAXa/7uyryLidHgrka09q0yw=="}' 2024-04-25T09:14:53.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d91c4f80-3723-41df-b7cb-c0ba7c928b9a -i td/pg-split-merge/0/new.json 2024-04-25T09:14:54.288 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:14:54.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:14:54.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANHypmqTeYOxAAXa/7uyryLidHgrka09q0yw== --osd-uuid d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:14:54.336 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.346+0000 7fd8609da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:54.356 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.366+0000 7fd8609da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:54.358 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.370+0000 7fd8609da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:54.358 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.370+0000 7fd8609da7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:14:54.358 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.370+0000 7fd8609da7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:14:54.358 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.370+0000 7fd8609da7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:14:54.387 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:14:54.398+0000 7fd8609da7c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:15:00.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:15:00.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:15:00.563 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:15:00.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:15:00.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:15:01.187 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:15:01.188 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:15:01.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:01.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:15:01.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:15:01.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:15:01.222 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:01.234+0000 7f1126cd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:01.226 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:01.238+0000 7f1126cd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:01.227 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:01.238+0000 7f1126cd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:01.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:15:01.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:01.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:15:01.656 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:15:01.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:01.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:01.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:01.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:01.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:01.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:01.790 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:01.802+0000 7f1126cd37c0 -1 Falling back to public interface 2024-04-25T09:15:02.096 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:03.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:03.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:03.099 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:15:03.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:03.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:03.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:03.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:04.328 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:04.338+0000 7f1126cd37c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:15:04.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:04.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:04.529 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:15:04.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:04.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:04.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:05.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:06.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:06.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:06.048 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:15:06.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:06.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:06.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:06.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:07.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:07.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:07.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:15:07.489 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:15:07.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:07.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:08.097 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:09.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:09.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:09.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:15:09.099 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:15:09.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:09.099 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:09.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:10.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:10.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:10.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:15:10.555 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:15:10.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:10.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:11.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:12.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:12.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:12.022 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:15:12.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:15:12.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:12.023 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:12.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:13.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:13.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:13.459 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:15:13.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:15:13.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:13.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:13.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:14.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:14.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:14.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:15:14.889 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:15:14.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:14.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:15.352 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:16.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:16.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:16.354 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:15:16.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:15:16.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:16.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:17.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:18.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:18.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:18.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:15:18.059 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:15:18.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:18.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:18.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:19.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:19.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:19.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:15:19.570 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:15:19.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:19.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:20.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:21.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:21.249 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:21.249 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:15:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:15:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:21.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:22.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:23.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:23.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:23.292 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:15:23.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:15:23.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:23.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:24.055 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:25.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:25.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:25.057 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:15:25.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:15:25.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:25.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:26.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:27.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:27.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:27.172 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:15:27.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:15:27.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:27.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:27.631 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3047933981,v1:127.0.0.1:6803/3047933981] [v2:127.0.0.1:6804/3047933981,v1:127.0.0.1:6805/3047933981] exists,up d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:15:27.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:15:27.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:15:27.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:15:27.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-04-25T09:15:28.157 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T09:15:28.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:15:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:15:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:15:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:15:28.175 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:15:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:15:28.176 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:15:28.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:15:28.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:15:28.176 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:15:28.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:15:28.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:15:28.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:15:28.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:15:28.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:15:28.282 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:15:28.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:15:28.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:15:28.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:28.807 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:15:29.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-04-25T09:15:29.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-04-25T09:15:29.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-04-25T09:15:29.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:29.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-04-25T09:15:29.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:29.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:15:29.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-04-25T09:15:29.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:29.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-04-25T09:15:29.064 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836493 2024-04-25T09:15:29.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-04-25T09:15:29.065 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:15:29.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836493 2024-04-25T09:15:29.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:15:30.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:15:30.550 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:15:30.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836493 2024-04-25T09:15:30.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:15:30.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:15:30.989 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:15:31.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:15:31.552 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:15:31.552 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:15:31.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:15:31.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:15:31.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:15:31.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:15:31.553 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:15:31.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:15:31.973 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:15:31.973 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:15:31.973 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:15:32.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:15:32.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:15:32.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:15:32.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:15:32.596 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T09:15:32.596 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:15:32.596 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_454752 2024-04-25T09:15:32.596 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:15:32.596 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:15:33.596 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 128 112 0.109362 0.109375 0.101105 0.135511 2024-04-25T09:15:34.596 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 273 257 0.12547 0.141602 0.117238 0.119458 2024-04-25T09:15:35.596 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 401 385 0.125306 0.125 0.117246 0.121169 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 3.07868 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 401 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 1024 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 1024 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.127198 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0161157 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.141602 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.109375 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 130 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 16.5025 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 145 2024-04-25T09:15:36.597 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 112 2024-04-25T09:15:36.598 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.120961 2024-04-25T09:15:36.598 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0305782 2024-04-25T09:15:36.598 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.210388 2024-04-25T09:15:36.598 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.0962214 2024-04-25T09:15:36.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:15:36.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:15:36.656 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:15:36.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:15:36.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:15:36.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:15:36.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:15:36.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:15:37.215 INFO:tasks.workunit.client.0.smithi031.stderr:Exporting 1.1 info 1.1( v 10'189 (0'0,10'189] local-lis/les=7/8 n=189 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:15:37.235 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:825fce9d:::benchmark_data_smithi031_454752_object9:head# 2024-04-25T09:15:37.251 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:82fe0341:::benchmark_data_smithi031_454752_object167:head# 2024-04-25T09:15:37.270 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:830b3926:::benchmark_data_smithi031_454752_object175:head# 2024-04-25T09:15:37.277 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:839ff2c1:::benchmark_data_smithi031_454752_object178:head# 2024-04-25T09:15:37.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:83bed9ab:::benchmark_data_smithi031_454752_object192:head# 2024-04-25T09:15:37.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:84bae7de:::benchmark_data_smithi031_454752_object19:head# 2024-04-25T09:15:37.286 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8514352c:::benchmark_data_smithi031_454752_object84:head# 2024-04-25T09:15:37.286 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:85e422cb:::benchmark_data_smithi031_454752_object183:head# 2024-04-25T09:15:37.287 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8696227e:::benchmark_data_smithi031_454752_object304:head# 2024-04-25T09:15:37.298 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:86aa36c1:::benchmark_data_smithi031_454752_object179:head# 2024-04-25T09:15:37.299 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:878ba262:::benchmark_data_smithi031_454752_object281:head# 2024-04-25T09:15:37.306 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:891fd6e5:::benchmark_data_smithi031_454752_object28:head# 2024-04-25T09:15:37.306 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8941388a:::benchmark_data_smithi031_454752_object87:head# 2024-04-25T09:15:37.310 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8a4773c4:::benchmark_data_smithi031_454752_object103:head# 2024-04-25T09:15:37.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8abbf253:::benchmark_data_smithi031_454752_object97:head# 2024-04-25T09:15:37.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8adf42cb:::benchmark_data_smithi031_454752_object180:head# 2024-04-25T09:15:37.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8ae888df:::benchmark_data_smithi031_454752_object382:head# 2024-04-25T09:15:37.313 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8b291374:::benchmark_data_smithi031_454752_object273:head# 2024-04-25T09:15:37.314 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8b621052:::benchmark_data_smithi031_454752_object368:head# 2024-04-25T09:15:37.316 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8c90b5b1:::benchmark_data_smithi031_454752_object161:head# 2024-04-25T09:15:37.317 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8d29fb8d:::benchmark_data_smithi031_454752_object327:head# 2024-04-25T09:15:37.317 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8db4e958:::benchmark_data_smithi031_454752_object385:head# 2024-04-25T09:15:37.318 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8fa26b27:::benchmark_data_smithi031_454752_object256:head# 2024-04-25T09:15:37.319 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:90588a1a:::benchmark_data_smithi031_454752_object378:head# 2024-04-25T09:15:37.320 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:907e8918:::benchmark_data_smithi031_454752_object266:head# 2024-04-25T09:15:37.320 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9277f059:::benchmark_data_smithi031_454752_object2:head# 2024-04-25T09:15:37.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:946e6356:::benchmark_data_smithi031_454752_object338:head# 2024-04-25T09:15:37.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9586170f:::benchmark_data_smithi031_454752_object13:head# 2024-04-25T09:15:37.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:95c08ad2:::benchmark_data_smithi031_454752_object210:head# 2024-04-25T09:15:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:95cd8cdf:::benchmark_data_smithi031_454752_object383:head# 2024-04-25T09:15:37.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9609476c:::benchmark_data_smithi031_454752_object114:head# 2024-04-25T09:15:37.327 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:967bc283:::benchmark_data_smithi031_454752_object231:head# 2024-04-25T09:15:37.330 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96801ae8:::benchmark_data_smithi031_454752_object207:head# 2024-04-25T09:15:37.331 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96a964d2:::benchmark_data_smithi031_454752_object38:head# 2024-04-25T09:15:37.331 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96aa787d:::benchmark_data_smithi031_454752_object322:head# 2024-04-25T09:15:37.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96d63edc:::benchmark_data_smithi031_454752_object352:head# 2024-04-25T09:15:37.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:98e8b5bf:::benchmark_data_smithi031_454752_object216:head# 2024-04-25T09:15:37.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:99e3add0:::benchmark_data_smithi031_454752_object5:head# 2024-04-25T09:15:37.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9a07c74e:::benchmark_data_smithi031_454752_object252:head# 2024-04-25T09:15:37.337 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9a72fd5f:::benchmark_data_smithi031_454752_object297:head# 2024-04-25T09:15:37.338 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9a9f4dbd:::benchmark_data_smithi031_454752_object321:head# 2024-04-25T09:15:37.340 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9b86cf51:::benchmark_data_smithi031_454752_object225:head# 2024-04-25T09:15:37.341 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9be87125:::benchmark_data_smithi031_454752_object116:head# 2024-04-25T09:15:37.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9bf0e6c9:::benchmark_data_smithi031_454752_object47:head# 2024-04-25T09:15:37.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9c836d36:::benchmark_data_smithi031_454752_object44:head# 2024-04-25T09:15:37.343 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9cb1d2e2:::benchmark_data_smithi031_454752_object362:head# 2024-04-25T09:15:37.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9cbe41c2:::benchmark_data_smithi031_454752_object374:head# 2024-04-25T09:15:37.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9ce31ece:::benchmark_data_smithi031_454752_object308:head# 2024-04-25T09:15:37.345 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9df3f789:::benchmark_data_smithi031_454752_object394:head# 2024-04-25T09:15:37.350 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9e952085:::benchmark_data_smithi031_454752_object154:head# 2024-04-25T09:15:37.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9f2a7ece:::benchmark_data_smithi031_454752_object251:head# 2024-04-25T09:15:37.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a114ad36:::benchmark_data_smithi031_454752_object70:head# 2024-04-25T09:15:37.353 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a419031b:::benchmark_data_smithi031_454752_object81:head# 2024-04-25T09:15:37.353 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a45f5752:::benchmark_data_smithi031_454752_object39:head# 2024-04-25T09:15:37.354 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a46e3cab:::benchmark_data_smithi031_454752_object193:head# 2024-04-25T09:15:37.354 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a46e59f0:::benchmark_data_smithi031_454752_object130:head# 2024-04-25T09:15:37.355 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a75374ab:::benchmark_data_smithi031_454752_object190:head# 2024-04-25T09:15:37.356 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a8271a67:::benchmark_data_smithi031_454752_object33:head# 2024-04-25T09:15:37.356 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a853f81a:::benchmark_data_smithi031_454752_object379:head# 2024-04-25T09:15:37.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a8c6bb46:::benchmark_data_smithi031_454752_object72:head# 2024-04-25T09:15:37.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a942e3ba:::benchmark_data_smithi031_454752_object74:head# 2024-04-25T09:15:37.358 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a98c4d7c:::benchmark_data_smithi031_454752_object37:head# 2024-04-25T09:15:37.358 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a9e307d2:::benchmark_data_smithi031_454752_object318:head# 2024-04-25T09:15:37.359 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aab383a6:::benchmark_data_smithi031_454752_object112:head# 2024-04-25T09:15:37.359 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aaf58300:::benchmark_data_smithi031_454752_object208:head# 2024-04-25T09:15:37.360 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab0f7dc2:::benchmark_data_smithi031_454752_object119:head# 2024-04-25T09:15:37.360 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab1cf81e:::benchmark_data_smithi031_454752_object263:head# 2024-04-25T09:15:37.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab49749c:::benchmark_data_smithi031_454752_object359:head# 2024-04-25T09:15:37.362 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ac1d5d1f:::benchmark_data_smithi031_454752_object135:head# 2024-04-25T09:15:37.362 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ac7ea5bf:::benchmark_data_smithi031_454752_object217:head# 2024-04-25T09:15:37.363 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:acc3e569:::benchmark_data_smithi031_454752_object290:head# 2024-04-25T09:15:37.364 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ad398d0f:::benchmark_data_smithi031_454752_object125:head# 2024-04-25T09:15:37.365 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ae91e1ef:::benchmark_data_smithi031_454752_object115:head# 2024-04-25T09:15:37.365 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aea3dfa5:::benchmark_data_smithi031_454752_object388:head# 2024-04-25T09:15:37.366 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aec2dd45:::benchmark_data_smithi031_454752_object275:head# 2024-04-25T09:15:37.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:af668e34:::benchmark_data_smithi031_454752_object302:head# 2024-04-25T09:15:37.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:af9a9573:::benchmark_data_smithi031_454752_object58:head# 2024-04-25T09:15:37.368 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b089bb4f:::benchmark_data_smithi031_454752_object218:head# 2024-04-25T09:15:37.368 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b1674de8:::benchmark_data_smithi031_454752_object194:head# 2024-04-25T09:15:37.369 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b2930ee9:::benchmark_data_smithi031_454752_object73:head# 2024-04-25T09:15:37.369 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b329d82c:::benchmark_data_smithi031_454752_object200:head# 2024-04-25T09:15:37.370 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b3c75192:::benchmark_data_smithi031_454752_object325:head# 2024-04-25T09:15:37.371 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b45e9911:::benchmark_data_smithi031_454752_object53:head# 2024-04-25T09:15:37.371 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b4f7e75f:::benchmark_data_smithi031_454752_object296:head# 2024-04-25T09:15:37.372 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b51eab52:::benchmark_data_smithi031_454752_object393:head# 2024-04-25T09:15:37.372 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b54079d2:::benchmark_data_smithi031_454752_object319:head# 2024-04-25T09:15:37.373 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b5427b66:::benchmark_data_smithi031_454752_object113:head# 2024-04-25T09:15:37.373 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b59f1134:::benchmark_data_smithi031_454752_object301:head# 2024-04-25T09:15:37.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b66ddae8:::benchmark_data_smithi031_454752_object206:head# 2024-04-25T09:15:37.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b6a31cad:::benchmark_data_smithi031_454752_object27:head# 2024-04-25T09:15:37.375 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b7073e4b:::benchmark_data_smithi031_454752_object126:head# 2024-04-25T09:15:37.375 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b83f72b1:::benchmark_data_smithi031_454752_object162:head# 2024-04-25T09:15:37.376 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:15:37.377 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b94a5f10:::benchmark_data_smithi031_454752_object348:head# 2024-04-25T09:15:37.378 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b9a0d9e0:::benchmark_data_smithi031_454752_object342:head# 2024-04-25T09:15:37.378 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb052c00:::benchmark_data_smithi031_454752_object88:head# 2024-04-25T09:15:37.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb48e6d2:::benchmark_data_smithi031_454752_object241:head# 2024-04-25T09:15:37.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb57b519:::benchmark_data_smithi031_454752_object50:head# 2024-04-25T09:15:37.380 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb6af9e7:::benchmark_data_smithi031_454752_object373:head# 2024-04-25T09:15:37.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bca0f028:::benchmark_data_smithi031_454752_object20:head# 2024-04-25T09:15:37.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bd00cd0f:::benchmark_data_smithi031_454752_object24:head# 2024-04-25T09:15:37.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bd3a993f:::benchmark_data_smithi031_454752_object255:head# 2024-04-25T09:15:37.382 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bdd858f4:::benchmark_data_smithi031_454752_object228:head# 2024-04-25T09:15:37.383 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bfa77210:::benchmark_data_smithi031_454752_object67:head# 2024-04-25T09:15:37.383 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c0619f0c:::benchmark_data_smithi031_454752_object247:head# 2024-04-25T09:15:37.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c0b886c3:::benchmark_data_smithi031_454752_object25:head# 2024-04-25T09:15:37.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c229294b:::benchmark_data_smithi031_454752_object127:head# 2024-04-25T09:15:37.385 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c2f8ad7b:::benchmark_data_smithi031_454752_object104:head# 2024-04-25T09:15:37.385 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c4fe132d:::benchmark_data_smithi031_454752_object152:head# 2024-04-25T09:15:37.386 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c5dc4341:::benchmark_data_smithi031_454752_object166:head# 2024-04-25T09:15:37.386 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c6367955:::benchmark_data_smithi031_454752_object144:head# 2024-04-25T09:15:37.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c664156b:::benchmark_data_smithi031_454752_object238:head# 2024-04-25T09:15:37.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c78167ae:::benchmark_data_smithi031_454752_object85:head# 2024-04-25T09:15:37.389 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c79d3970:::benchmark_data_smithi031_454752_object221:head# 2024-04-25T09:15:37.390 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c8eb0827:::benchmark_data_smithi031_454752_object298:head# 2024-04-25T09:15:37.390 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca1d51a2:::benchmark_data_smithi031_454752_object280:head# 2024-04-25T09:15:37.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cab67049:::benchmark_data_smithi031_454752_object91:head# 2024-04-25T09:15:37.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cad688fc:::benchmark_data_smithi031_454752_object351:head# 2024-04-25T09:15:37.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:caf98509:::benchmark_data_smithi031_454752_object196:head# 2024-04-25T09:15:37.393 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cc58c178:::benchmark_data_smithi031_454752_object32:head# 2024-04-25T09:15:37.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cc9c514e:::benchmark_data_smithi031_454752_object65:head# 2024-04-25T09:15:37.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cca29ffb:::benchmark_data_smithi031_454752_object80:head# 2024-04-25T09:15:37.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ccbd2f1f:::benchmark_data_smithi031_454752_object184:head# 2024-04-25T09:15:37.395 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cd8693fa:::benchmark_data_smithi031_454752_object398:head# 2024-04-25T09:15:37.396 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ceb870b0:::benchmark_data_smithi031_454752_object222:head# 2024-04-25T09:15:37.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d093eb2e:::benchmark_data_smithi031_454752_object311:head# 2024-04-25T09:15:37.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d231d661:::benchmark_data_smithi031_454752_object177:head# 2024-04-25T09:15:37.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d2ae4d7f:::benchmark_data_smithi031_454752_object51:head# 2024-04-25T09:15:37.398 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d35c9cea:::benchmark_data_smithi031_454752_object317:head# 2024-04-25T09:15:37.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d4438560:::benchmark_data_smithi031_454752_object143:head# 2024-04-25T09:15:37.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d452fc99:::benchmark_data_smithi031_454752_object45:head# 2024-04-25T09:15:37.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d6574e4a:::benchmark_data_smithi031_454752_object60:head# 2024-04-25T09:15:37.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d73435c2:::benchmark_data_smithi031_454752_object118:head# 2024-04-25T09:15:37.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d75560e8:::benchmark_data_smithi031_454752_object128:head# 2024-04-25T09:15:37.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d75a04bb:::benchmark_data_smithi031_454752_object98:head# 2024-04-25T09:15:37.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d75b7183:::benchmark_data_smithi031_454752_object230:head# 2024-04-25T09:15:37.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d7bf5e5b:::benchmark_data_smithi031_454752_object239:head# 2024-04-25T09:15:37.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d824c896:::benchmark_data_smithi031_454752_object286:head# 2024-04-25T09:15:37.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:da211111:::benchmark_data_smithi031_454752_object159:head# 2024-04-25T09:15:37.404 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:da25a4ff:::benchmark_data_smithi031_454752_object75:head# 2024-04-25T09:15:37.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:db18fece:::benchmark_data_smithi031_454752_object250:head# 2024-04-25T09:15:37.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dc4325f0:::benchmark_data_smithi031_454752_object131:head# 2024-04-25T09:15:37.406 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dedcc3de:::benchmark_data_smithi031_454752_object148:head# 2024-04-25T09:15:37.407 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:deec6ff8:::benchmark_data_smithi031_454752_object224:head# 2024-04-25T09:15:37.407 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e0881e69:::benchmark_data_smithi031_454752_object293:head# 2024-04-25T09:15:37.408 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e1341170:::benchmark_data_smithi031_454752_object132:head# 2024-04-25T09:15:37.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e17ed360:::benchmark_data_smithi031_454752_object140:head# 2024-04-25T09:15:37.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e3be378c:::benchmark_data_smithi031_454752_object188:head# 2024-04-25T09:15:37.410 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e3c37152:::benchmark_data_smithi031_454752_object155:head# 2024-04-25T09:15:37.411 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e431150e:::benchmark_data_smithi031_454752_object309:head# 2024-04-25T09:15:37.411 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e44b8a2c:::benchmark_data_smithi031_454752_object146:head# 2024-04-25T09:15:37.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e4a3dd60:::benchmark_data_smithi031_454752_object341:head# 2024-04-25T09:15:37.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e53228e8:::benchmark_data_smithi031_454752_object364:head# 2024-04-25T09:15:37.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e53430a0:::benchmark_data_smithi031_454752_object336:head# 2024-04-25T09:15:37.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e5497360:::benchmark_data_smithi031_454752_object141:head# 2024-04-25T09:15:37.414 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e5ae028f:::benchmark_data_smithi031_454752_object254:head# 2024-04-25T09:15:37.414 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e72dd955:::benchmark_data_smithi031_454752_object105:head# 2024-04-25T09:15:37.415 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e75527e1:::benchmark_data_smithi031_454752_object365:head# 2024-04-25T09:15:37.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e8e4cd11:::benchmark_data_smithi031_454752_object171:head# 2024-04-25T09:15:37.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e94a0572:::benchmark_data_smithi031_454752_object6:head# 2024-04-25T09:15:37.417 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e991363a:::benchmark_data_smithi031_454752_object52:head# 2024-04-25T09:15:37.417 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ea70172b:::benchmark_data_smithi031_454752_object191:head# 2024-04-25T09:15:37.418 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:eaf3d89c:::benchmark_data_smithi031_454752_object358:head# 2024-04-25T09:15:37.418 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ec070df6:::benchmark_data_smithi031_454752_object234:head# 2024-04-25T09:15:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ec82b70c:::benchmark_data_smithi031_454752_object246:head# 2024-04-25T09:15:37.419 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ed17f3a5:::benchmark_data_smithi031_454752_object389:head# 2024-04-25T09:15:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:edd73bf9:::benchmark_data_smithi031_454752_object95:head# 2024-04-25T09:15:37.420 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ee0f64f0:::benchmark_data_smithi031_454752_object49:head# 2024-04-25T09:15:37.421 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f054ad11:::benchmark_data_smithi031_454752_object145:head# 2024-04-25T09:15:37.421 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f102181e:::benchmark_data_smithi031_454752_object288:head# 2024-04-25T09:15:37.422 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f21d79de:::benchmark_data_smithi031_454752_object66:head# 2024-04-25T09:15:37.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f2b747ee:::benchmark_data_smithi031_454752_object314:head# 2024-04-25T09:15:37.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f34b8ad2:::benchmark_data_smithi031_454752_object240:head# 2024-04-25T09:15:37.424 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f4863048:::benchmark_data_smithi031_454752_object349:head# 2024-04-25T09:15:37.424 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f4ab8147:::benchmark_data_smithi031_454752_object337:head# 2024-04-25T09:15:37.425 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f547a4d2:::benchmark_data_smithi031_454752_object369:head# 2024-04-25T09:15:37.425 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f6236fee:::benchmark_data_smithi031_454752_object289:head# 2024-04-25T09:15:37.426 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f635f136:::benchmark_data_smithi031_454752_object23:head# 2024-04-25T09:15:37.427 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f6e629fe:::benchmark_data_smithi031_454752_object57:head# 2024-04-25T09:15:37.427 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f70fb177:::benchmark_data_smithi031_454752_object265:head# 2024-04-25T09:15:37.428 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f77401b8:::benchmark_data_smithi031_454752_object71:head# 2024-04-25T09:15:37.428 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f870ff0e:::benchmark_data_smithi031_454752_object312:head# 2024-04-25T09:15:37.429 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:faefa441:::benchmark_data_smithi031_454752_object269:head# 2024-04-25T09:15:37.429 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb1204b7:::benchmark_data_smithi031_454752_object331:head# 2024-04-25T09:15:37.430 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb3355e2:::benchmark_data_smithi031_454752_object363:head# 2024-04-25T09:15:37.430 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb5c543a:::benchmark_data_smithi031_454752_object329:head# 2024-04-25T09:15:37.431 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fcf4159f:::benchmark_data_smithi031_454752_object185:head# 2024-04-25T09:15:37.431 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fe7c790a:::benchmark_data_smithi031_454752_object16:head# 2024-04-25T09:15:37.432 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ffeaaa0c:::benchmark_data_smithi031_454752_object139:head# 2024-04-25T09:15:37.432 INFO:tasks.workunit.client.0.smithi031.stderr:Export successful 2024-04-25T09:15:37.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:15:37.834 INFO:tasks.workunit.client.0.smithi031.stderr:Exporting 1.0 info 1.0( v 10'213 (0'0,10'213] local-lis/les=7/8 n=213 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:15:37.837 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0005830b:::benchmark_data_smithi031_454752_object169:head# 2024-04-25T09:15:37.837 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0007ff59:::benchmark_data_smithi031_454752_object384:head# 2024-04-25T09:15:37.837 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00530cad:::benchmark_data_smithi031_454752_object366:head# 2024-04-25T09:15:37.838 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:007182ad:::benchmark_data_smithi031_454752_object150:head# 2024-04-25T09:15:37.838 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00808219:::benchmark_data_smithi031_454752_object10:head# 2024-04-25T09:15:37.838 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00ccfbba:::benchmark_data_smithi031_454752_object355:head# 2024-04-25T09:15:37.839 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00d7eda8:::benchmark_data_smithi031_454752_object357:head# 2024-04-25T09:15:37.839 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:01faf234:::benchmark_data_smithi031_454752_object272:head# 2024-04-25T09:15:37.840 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:021d5fe5:::benchmark_data_smithi031_454752_object109:head# 2024-04-25T09:15:37.840 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:02a87b95:::benchmark_data_smithi031_454752_object26:head# 2024-04-25T09:15:37.840 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:03119e67:::benchmark_data_smithi031_454752_object120:head# 2024-04-25T09:15:37.840 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:03a19a01:::benchmark_data_smithi031_454752_object375:head# 2024-04-25T09:15:37.841 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:03d8d0d6:::benchmark_data_smithi031_454752_object307:head# 2024-04-25T09:15:37.841 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0407316c:::benchmark_data_smithi031_454752_object248:head# 2024-04-25T09:15:37.842 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:069ac774:::benchmark_data_smithi031_454752_object270:head# 2024-04-25T09:15:37.842 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:076ac3b3:::benchmark_data_smithi031_454752_object168:head# 2024-04-25T09:15:37.842 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:078ca8f4:::benchmark_data_smithi031_454752_object271:head# 2024-04-25T09:15:37.843 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:089737ee:::benchmark_data_smithi031_454752_object262:head# 2024-04-25T09:15:37.843 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0954f670:::benchmark_data_smithi031_454752_object223:head# 2024-04-25T09:15:37.843 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:09d319a2:::benchmark_data_smithi031_454752_object283:head# 2024-04-25T09:15:37.844 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0a33ee0a:::benchmark_data_smithi031_454752_object107:head# 2024-04-25T09:15:37.845 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0a588e5f:::benchmark_data_smithi031_454752_object305:head# 2024-04-25T09:15:37.845 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ae0c865:::benchmark_data_smithi031_454752_object41:head# 2024-04-25T09:15:37.845 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0c38e0a3:::benchmark_data_smithi031_454752_object387:head# 2024-04-25T09:15:37.845 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0c4decad:::benchmark_data_smithi031_454752_object151:head# 2024-04-25T09:15:37.846 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0cc7bcb7:::benchmark_data_smithi031_454752_object333:head# 2024-04-25T09:15:37.847 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ce7b7bc:::benchmark_data_smithi031_454752_object353:head# 2024-04-25T09:15:37.847 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0d6e3436:::benchmark_data_smithi031_454752_object205:head# 2024-04-25T09:15:37.847 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0dd158a0:::benchmark_data_smithi031_454752_object76:head# 2024-04-25T09:15:37.847 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0df905cb:::benchmark_data_smithi031_454752_object181:head# 2024-04-25T09:15:37.848 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0f047668:::benchmark_data_smithi031_454752_object12:head# 2024-04-25T09:15:37.848 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:100b6e67:::benchmark_data_smithi031_454752_object122:head# 2024-04-25T09:15:37.848 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:105e9b29:::benchmark_data_smithi031_454752_object295:head# 2024-04-25T09:15:37.848 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:10ca342c:::benchmark_data_smithi031_454752_object203:head# 2024-04-25T09:15:37.849 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:11985b73:::benchmark_data_smithi031_454752_object294:head# 2024-04-25T09:15:37.849 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1215ebb0:::benchmark_data_smithi031_454752_object96:head# 2024-04-25T09:15:37.850 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:123f25d6:::benchmark_data_smithi031_454752_object339:head# 2024-04-25T09:15:37.850 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:12648b62:::benchmark_data_smithi031_454752_object235:head# 2024-04-25T09:15:37.850 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:13b5933b:::benchmark_data_smithi031_454752_object258:head# 2024-04-25T09:15:37.851 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:14a4f03a:::benchmark_data_smithi031_454752_object90:head# 2024-04-25T09:15:37.851 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:152756a2:::benchmark_data_smithi031_454752_object78:head# 2024-04-25T09:15:37.851 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:17685eef:::benchmark_data_smithi031_454752_object35:head# 2024-04-25T09:15:37.852 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:17d78309:::benchmark_data_smithi031_454752_object197:head# 2024-04-25T09:15:37.852 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:186323b7:::benchmark_data_smithi031_454752_object264:head# 2024-04-25T09:15:37.852 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18a96ee0:::benchmark_data_smithi031_454752_object340:head# 2024-04-25T09:15:37.853 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18abc847:::benchmark_data_smithi031_454752_object377:head# 2024-04-25T09:15:37.853 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:190ad911:::benchmark_data_smithi031_454752_object158:head# 2024-04-25T09:15:37.853 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:192b4b6c:::benchmark_data_smithi031_454752_object249:head# 2024-04-25T09:15:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1a2dbf6a:::benchmark_data_smithi031_454752_object284:head# 2024-04-25T09:15:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1bace3cc:::benchmark_data_smithi031_454752_object136:head# 2024-04-25T09:15:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1f6172c3:::benchmark_data_smithi031_454752_object315:head# 2024-04-25T09:15:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:20988891:::benchmark_data_smithi031_454752_object0:head# 2024-04-25T09:15:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:20c89311:::benchmark_data_smithi031_454752_object170:head# 2024-04-25T09:15:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:20f96ee6:::benchmark_data_smithi031_454752_object30:head# 2024-04-25T09:15:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:210b81be:::benchmark_data_smithi031_454752_object396:head# 2024-04-25T09:15:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:222bd3b2:::benchmark_data_smithi031_454752_object244:head# 2024-04-25T09:15:37.855 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:23ea9d14:::benchmark_data_smithi031_454752_object134:head# 2024-04-25T09:15:37.856 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2564bee6:::benchmark_data_smithi031_454752_object68:head# 2024-04-25T09:15:37.856 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:26e5d41e:::benchmark_data_smithi031_454752_object261:head# 2024-04-25T09:15:37.857 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2765dc2c:::benchmark_data_smithi031_454752_object147:head# 2024-04-25T09:15:37.857 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:286a6080:::benchmark_data_smithi031_454752_object64:head# 2024-04-25T09:15:37.858 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2899a918:::benchmark_data_smithi031_454752_object267:head# 2024-04-25T09:15:37.858 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:299c9ca7:::benchmark_data_smithi031_454752_object21:head# 2024-04-25T09:15:37.858 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:29db69da:::benchmark_data_smithi031_454752_object236:head# 2024-04-25T09:15:37.858 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2aa178ac:::benchmark_data_smithi031_454752_object59:head# 2024-04-25T09:15:37.858 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2ac07948:::benchmark_data_smithi031_454752_object7:head# 2024-04-25T09:15:37.858 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2af7345c:::benchmark_data_smithi031_454752_object202:head# 2024-04-25T09:15:37.859 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2c7cae34:::benchmark_data_smithi031_454752_object300:head# 2024-04-25T09:15:37.859 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2d714369:::benchmark_data_smithi031_454752_object292:head# 2024-04-25T09:15:37.859 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2da27532:::benchmark_data_smithi031_454752_object1:head# 2024-04-25T09:15:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2dee9738:::benchmark_data_smithi031_454752_object400:head# 2024-04-25T09:15:37.860 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2e61c031:::benchmark_data_smithi031_454752_object174:head# 2024-04-25T09:15:37.861 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2ee4a5f6:::benchmark_data_smithi031_454752_object94:head# 2024-04-25T09:15:37.861 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2f1e460a:::benchmark_data_smithi031_454752_object106:head# 2024-04-25T09:15:37.861 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2fc8a11e:::benchmark_data_smithi031_454752_object260:head# 2024-04-25T09:15:37.862 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:30797560:::benchmark_data_smithi031_454752_object142:head# 2024-04-25T09:15:37.862 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3105618d:::benchmark_data_smithi031_454752_object46:head# 2024-04-25T09:15:37.863 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31a53b00:::benchmark_data_smithi031_454752_object14:head# 2024-04-25T09:15:37.863 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31b8e0ab:::benchmark_data_smithi031_454752_object326:head# 2024-04-25T09:15:37.863 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31c961a5:::benchmark_data_smithi031_454752_object11:head# 2024-04-25T09:15:37.863 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31f4feb6:::benchmark_data_smithi031_454752_object54:head# 2024-04-25T09:15:37.863 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:332a7b27:::benchmark_data_smithi031_454752_object257:head# 2024-04-25T09:15:37.864 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:33cf1783:::benchmark_data_smithi031_454752_object83:head# 2024-04-25T09:15:37.864 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:33e69519:::benchmark_data_smithi031_454752_object334:head# 2024-04-25T09:15:37.864 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:343b464c:::benchmark_data_smithi031_454752_object229:head# 2024-04-25T09:15:37.865 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:353bc8d6:::benchmark_data_smithi031_454752_object306:head# 2024-04-25T09:15:37.865 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:357adf2e:::benchmark_data_smithi031_454752_object313:head# 2024-04-25T09:15:37.865 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3626a3d2:::benchmark_data_smithi031_454752_object242:head# 2024-04-25T09:15:37.866 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:366a3868:::benchmark_data_smithi031_454752_object344:head# 2024-04-25T09:15:37.866 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:36e1e4b7:::benchmark_data_smithi031_454752_object330:head# 2024-04-25T09:15:37.867 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3747f9e2:::benchmark_data_smithi031_454752_object361:head# 2024-04-25T09:15:37.867 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:376c5fd1:::benchmark_data_smithi031_454752_object276:head# 2024-04-25T09:15:37.868 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:37ad6f44:::benchmark_data_smithi031_454752_object199:head# 2024-04-25T09:15:37.868 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3809a39d:::benchmark_data_smithi031_454752_object63:head# 2024-04-25T09:15:37.868 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:38962184:::benchmark_data_smithi031_454752_object345:head# 2024-04-25T09:15:37.869 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3947aa59:::benchmark_data_smithi031_454752_object29:head# 2024-04-25T09:15:37.869 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:39a62bf7:::benchmark_data_smithi031_454752_object346:head# 2024-04-25T09:15:37.870 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3a70fad2:::benchmark_data_smithi031_454752_object213:head# 2024-04-25T09:15:37.870 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3aaccaa2:::benchmark_data_smithi031_454752_object245:head# 2024-04-25T09:15:37.870 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3b4d69f0:::benchmark_data_smithi031_454752_object133:head# 2024-04-25T09:15:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3b5ed4ad:::benchmark_data_smithi031_454752_object153:head# 2024-04-25T09:15:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3bd7c293:::benchmark_data_smithi031_454752_object226:head# 2024-04-25T09:15:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3d9bbce6:::benchmark_data_smithi031_454752_object195:head# 2024-04-25T09:15:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3e1dceb1:::benchmark_data_smithi031_454752_object163:head# 2024-04-25T09:15:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3f585de7:::benchmark_data_smithi031_454752_object370:head# 2024-04-25T09:15:37.872 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3f8dddc0:::benchmark_data_smithi031_454752_object36:head# 2024-04-25T09:15:37.872 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:40e869df:::benchmark_data_smithi031_454752_object354:head# 2024-04-25T09:15:37.873 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:42f70b52:::benchmark_data_smithi031_454752_object391:head# 2024-04-25T09:15:37.873 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:44648b7f:::benchmark_data_smithi031_454752_object15:head# 2024-04-25T09:15:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:45c0069e:::benchmark_data_smithi031_454752_object93:head# 2024-04-25T09:15:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:46cce1fb:::benchmark_data_smithi031_454752_object259:head# 2024-04-25T09:15:37.874 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:48372743:::benchmark_data_smithi031_454752_object92:head# 2024-04-25T09:15:37.875 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:487de23f:::benchmark_data_smithi031_454752_object278:head# 2024-04-25T09:15:37.875 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:48933b4f:::benchmark_data_smithi031_454752_object219:head# 2024-04-25T09:15:37.875 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:48b43814:::benchmark_data_smithi031_454752_object22:head# 2024-04-25T09:15:37.876 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:49675aeb:::benchmark_data_smithi031_454752_object99:head# 2024-04-25T09:15:37.876 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a2d5412:::benchmark_data_smithi031_454752_object324:head# 2024-04-25T09:15:37.877 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a6e0d41:::benchmark_data_smithi031_454752_object268:head# 2024-04-25T09:15:37.877 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a6e92f8:::benchmark_data_smithi031_454752_object198:head# 2024-04-25T09:15:37.877 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a95c670:::benchmark_data_smithi031_454752_object220:head# 2024-04-25T09:15:37.878 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a9a32d5:::benchmark_data_smithi031_454752_object56:head# 2024-04-25T09:15:37.878 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4aa2f1b1:::benchmark_data_smithi031_454752_object160:head# 2024-04-25T09:15:37.878 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4ac6a300:::benchmark_data_smithi031_454752_object209:head# 2024-04-25T09:15:37.879 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b065b88:::benchmark_data_smithi031_454752_object34:head# 2024-04-25T09:15:37.879 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b3ae5fa:::benchmark_data_smithi031_454752_object399:head# 2024-04-25T09:15:37.879 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b491b4d:::benchmark_data_smithi031_454752_object86:head# 2024-04-25T09:15:37.880 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4c281e67:::benchmark_data_smithi031_454752_object123:head# 2024-04-25T09:15:37.880 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cd1274c:::benchmark_data_smithi031_454752_object77:head# 2024-04-25T09:15:37.880 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cd91fcf:::benchmark_data_smithi031_454752_object227:head# 2024-04-25T09:15:37.880 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cd9e311:::benchmark_data_smithi031_454752_object165:head# 2024-04-25T09:15:37.881 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4e704b66:::benchmark_data_smithi031_454752_object110:head# 2024-04-25T09:15:37.881 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4f141352:::benchmark_data_smithi031_454752_object390:head# 2024-04-25T09:15:37.881 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4f67372e:::benchmark_data_smithi031_454752_object310:head# 2024-04-25T09:15:37.882 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5055bf66:::benchmark_data_smithi031_454752_object111:head# 2024-04-25T09:15:37.882 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5079fdc4:::benchmark_data_smithi031_454752_object102:head# 2024-04-25T09:15:37.882 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51592368:::benchmark_data_smithi031_454752_object129:head# 2024-04-25T09:15:37.883 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51c35831:::benchmark_data_smithi031_454752_object277:head# 2024-04-25T09:15:37.883 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:525ef634:::benchmark_data_smithi031_454752_object303:head# 2024-04-25T09:15:37.884 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:52f1a683:::benchmark_data_smithi031_454752_object232:head# 2024-04-25T09:15:37.884 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:549943cc:::benchmark_data_smithi031_454752_object137:head# 2024-04-25T09:15:37.885 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:54a7cad2:::benchmark_data_smithi031_454752_object243:head# 2024-04-25T09:15:37.885 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56c9c8a3:::benchmark_data_smithi031_454752_object386:head# 2024-04-25T09:15:37.885 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56d22a80:::benchmark_data_smithi031_454752_object61:head# 2024-04-25T09:15:37.885 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:576c3dbd:::benchmark_data_smithi031_454752_object320:head# 2024-04-25T09:15:37.886 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:580c5a46:::benchmark_data_smithi031_454752_object62:head# 2024-04-25T09:15:37.886 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:58815a7c:::benchmark_data_smithi031_454752_object392:head# 2024-04-25T09:15:37.886 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:58e6f983:::benchmark_data_smithi031_454752_object233:head# 2024-04-25T09:15:37.887 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:59c109e7:::benchmark_data_smithi031_454752_object372:head# 2024-04-25T09:15:37.887 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5a7c4e16:::benchmark_data_smithi031_454752_object48:head# 2024-04-25T09:15:37.888 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5a943612:::benchmark_data_smithi031_454752_object40:head# 2024-04-25T09:15:37.888 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5aed2644:::benchmark_data_smithi031_454752_object395:head# 2024-04-25T09:15:37.888 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5c81318e:::benchmark_data_smithi031_454752_object18:head# 2024-04-25T09:15:37.889 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5d21802c:::benchmark_data_smithi031_454752_object201:head# 2024-04-25T09:15:37.889 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5d50b842:::benchmark_data_smithi031_454752_object282:head# 2024-04-25T09:15:37.889 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5de1e511:::benchmark_data_smithi031_454752_object173:head# 2024-04-25T09:15:37.889 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e1e382c:::benchmark_data_smithi031_454752_object189:head# 2024-04-25T09:15:37.890 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e3603a8:::benchmark_data_smithi031_454752_object356:head# 2024-04-25T09:15:37.890 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e9a3526:::benchmark_data_smithi031_454752_object274:head# 2024-04-25T09:15:37.891 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:60dee496:::benchmark_data_smithi031_454752_object287:head# 2024-04-25T09:15:37.891 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:610072df:::benchmark_data_smithi031_454752_object381:head# 2024-04-25T09:15:37.891 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:61757bc4:::benchmark_data_smithi031_454752_object100:head# 2024-04-25T09:15:37.892 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:61eb06a1:::benchmark_data_smithi031_454752_object42:head# 2024-04-25T09:15:37.892 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:622a095a:::benchmark_data_smithi031_454752_object172:head# 2024-04-25T09:15:37.893 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:629696e0:::benchmark_data_smithi031_454752_object343:head# 2024-04-25T09:15:37.893 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:62dff125:::benchmark_data_smithi031_454752_object117:head# 2024-04-25T09:15:37.893 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:637e2a8b:::benchmark_data_smithi031_454752_object124:head# 2024-04-25T09:15:37.894 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:63a84c81:::benchmark_data_smithi031_454752_object82:head# 2024-04-25T09:15:37.895 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:64a1f0ad:::benchmark_data_smithi031_454752_object367:head# 2024-04-25T09:15:37.895 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:65a082ea:::benchmark_data_smithi031_454752_object316:head# 2024-04-25T09:15:37.895 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:66d0a0c5:::benchmark_data_smithi031_454752_object4:head# 2024-04-25T09:15:37.895 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6757b3f7:::benchmark_data_smithi031_454752_object347:head# 2024-04-25T09:15:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:68851dbf:::benchmark_data_smithi031_454752_object279:head# 2024-04-25T09:15:37.896 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:68869d9f:::benchmark_data_smithi031_454752_object8:head# 2024-04-25T09:15:37.897 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6922a669:::benchmark_data_smithi031_454752_object291:head# 2024-04-25T09:15:37.897 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:695ddfd2:::benchmark_data_smithi031_454752_object285:head# 2024-04-25T09:15:37.897 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:69f748df:::benchmark_data_smithi031_454752_object380:head# 2024-04-25T09:15:37.898 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6a4d4ed2:::benchmark_data_smithi031_454752_object212:head# 2024-04-25T09:15:37.898 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6d7df3bd:::benchmark_data_smithi031_454752_object323:head# 2024-04-25T09:15:37.899 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6d9bcab7:::benchmark_data_smithi031_454752_object299:head# 2024-04-25T09:15:37.899 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6e1dd161:::benchmark_data_smithi031_454752_object176:head# 2024-04-25T09:15:37.900 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ea26cda:::benchmark_data_smithi031_454752_object237:head# 2024-04-25T09:15:37.900 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:715f84d9:::benchmark_data_smithi031_454752_object31:head# 2024-04-25T09:15:37.901 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:718b1888:::benchmark_data_smithi031_454752_object3:head# 2024-04-25T09:15:37.901 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7201ba57:::benchmark_data_smithi031_454752_object332:head# 2024-04-25T09:15:37.901 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72916a4f:::benchmark_data_smithi031_454752_object43:head# 2024-04-25T09:15:37.902 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72e5e7bc:::benchmark_data_smithi031_454752_object350:head# 2024-04-25T09:15:37.902 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72f2ccd2:::benchmark_data_smithi031_454752_object211:head# 2024-04-25T09:15:37.902 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:737dcd92:::benchmark_data_smithi031_454752_object335:head# 2024-04-25T09:15:37.903 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:744eb9c4:::benchmark_data_smithi031_454752_object101:head# 2024-04-25T09:15:37.903 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:75451e3a:::benchmark_data_smithi031_454752_object215:head# 2024-04-25T09:15:37.904 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:75b87d24:::benchmark_data_smithi031_454752_object156:head# 2024-04-25T09:15:37.904 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:76a44be7:::benchmark_data_smithi031_454752_object371:head# 2024-04-25T09:15:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:76a5bace:::benchmark_data_smithi031_454752_object253:head# 2024-04-25T09:15:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:772eee67:::benchmark_data_smithi031_454752_object121:head# 2024-04-25T09:15:37.905 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:798c5654:::benchmark_data_smithi031_454752_object204:head# 2024-04-25T09:15:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7995db24:::benchmark_data_smithi031_454752_object157:head# 2024-04-25T09:15:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79abc611:::benchmark_data_smithi031_454752_object164:head# 2024-04-25T09:15:37.906 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79d1490c:::benchmark_data_smithi031_454752_object138:head# 2024-04-25T09:15:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7b6edc17:::benchmark_data_smithi031_454752_object214:head# 2024-04-25T09:15:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7b93ca23:::benchmark_data_smithi031_454752_object55:head# 2024-04-25T09:15:37.907 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bd5320a:::benchmark_data_smithi031_454752_object328:head# 2024-04-25T09:15:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bde86cb:::benchmark_data_smithi031_454752_object182:head# 2024-04-25T09:15:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bfed9be:::benchmark_data_smithi031_454752_object397:head# 2024-04-25T09:15:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7c0b3015:::benchmark_data_smithi031_454752_object108:head# 2024-04-25T09:15:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7cb420f8:::benchmark_data_smithi031_454752_object89:head# 2024-04-25T09:15:37.909 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7d588f87:::benchmark_data_smithi031_454752_object376:head# 2024-04-25T09:15:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dbcb22e:::benchmark_data_smithi031_454752_object17:head# 2024-04-25T09:15:37.910 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dc6b3de:::benchmark_data_smithi031_454752_object187:head# 2024-04-25T09:15:37.911 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dec0b89:::benchmark_data_smithi031_454752_object149:head# 2024-04-25T09:15:37.911 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dfe5ddd:::benchmark_data_smithi031_454752_object79:head# 2024-04-25T09:15:37.912 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7ea5a1e2:::benchmark_data_smithi031_454752_object360:head# 2024-04-25T09:15:37.912 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7eeb5bde:::benchmark_data_smithi031_454752_object186:head# 2024-04-25T09:15:37.913 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7fb12f34:::benchmark_data_smithi031_454752_object69:head# 2024-04-25T09:15:37.913 INFO:tasks.workunit.client.0.smithi031.stderr:Export successful 2024-04-25T09:15:38.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:15:38.204 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:15:38.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:15:38.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:15:38.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:15:38.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:15:38.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:38.205 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:15:38.207 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:15:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:15:38.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:15:38.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:15:38.209 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:15:38.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:15:38.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:38.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:15:38.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:15:38.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:15:38.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:15:38.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:15:38.246 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:38.257+0000 7f4ea9d1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:38.248 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:38.261+0000 7f4ea9d1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:38.250 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:38.261+0000 7f4ea9d1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:38.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:15:38.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:38.693 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:38.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:38.818 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:38.829+0000 7f4ea9d1b7c0 -1 Falling back to public interface 2024-04-25T09:15:39.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:40.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:40.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:40.247 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:15:40.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:40.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:40.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:40.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:41.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:41.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:41.690 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:15:41.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:41.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:41.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:41.821 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:15:41.833+0000 7f4ea9d1b7c0 -1 osd.0 10 log_to_monitors true 2024-04-25T09:15:42.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:43.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:43.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:43.157 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:15:43.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:43.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:43.158 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:43.645 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/553544540,v1:127.0.0.1:6803/553544540] [v2:127.0.0.1:6804/553544540,v1:127.0.0.1:6805/553544540] exists,up d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:15:43.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:15:43.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:15:43.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:15:43.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-04-25T09:15:44.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:15:49.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-04-25T09:15:49.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-04-25T09:15:49.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-04-25T09:15:49.503 INFO:tasks.workunit.client.0.smithi031.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:15:49.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:15:49.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:15:49.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:15:49.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:15:49.505 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:15:49.506 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:15:49.506 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:15:49.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:15:49.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:15:49.506 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:15:49.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:15:49.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:15:49.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:15:49.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:15:49.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:15:49.610 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:15:50.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:15:50.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:15:50.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:50.077 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:15:50.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T09:15:50.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T09:15:50.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T09:15:50.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:50.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:50.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T09:15:50.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:15:50.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T09:15:50.273 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:50.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T09:15:50.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T09:15:50.274 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 55834574852 2024-04-25T09:15:50.275 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:15:50.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574850 -lt 55834574852 2024-04-25T09:15:50.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:15:51.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:15:51.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:15:52.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T09:15:52.157 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:15:52.157 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:15:52.157 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:15:52.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:15:52.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:15:52.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:15:52.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:15:52.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:15:52.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:15:52.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:15:52.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:15:53.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:15:53.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:15:53.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:15:53.136 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:15:53.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:15:53.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:15:53.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:15:53.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:15:53.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:15:53.672 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:15:53.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:15:53.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:15:53.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:15:53.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:15:53.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:15:55.155 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:15:55.156 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:15:55.156 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:15:55.739 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T09:15:56.191 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:15:57.117 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e21 pg_nums {1={7=2,20=1}} deleted_pools ) 2024-04-25T09:15:57.118 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.1 2024-04-25T09:15:57.118 INFO:tasks.workunit.client.0.smithi031.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:15:57.119 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:825fce9d:::benchmark_data_smithi031_454752_object9:head# 2024-04-25T09:15:57.119 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.192 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:82fe0341:::benchmark_data_smithi031_454752_object167:head# 2024-04-25T09:15:57.192 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.226 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:830b3926:::benchmark_data_smithi031_454752_object175:head# 2024-04-25T09:15:57.226 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.259 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:839ff2c1:::benchmark_data_smithi031_454752_object178:head# 2024-04-25T09:15:57.259 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:83bed9ab:::benchmark_data_smithi031_454752_object192:head# 2024-04-25T09:15:57.301 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.334 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:84bae7de:::benchmark_data_smithi031_454752_object19:head# 2024-04-25T09:15:57.335 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.368 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8514352c:::benchmark_data_smithi031_454752_object84:head# 2024-04-25T09:15:57.368 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.401 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:85e422cb:::benchmark_data_smithi031_454752_object183:head# 2024-04-25T09:15:57.401 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.443 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8696227e:::benchmark_data_smithi031_454752_object304:head# 2024-04-25T09:15:57.443 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.477 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:86aa36c1:::benchmark_data_smithi031_454752_object179:head# 2024-04-25T09:15:57.477 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.510 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:878ba262:::benchmark_data_smithi031_454752_object281:head# 2024-04-25T09:15:57.510 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.544 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:891fd6e5:::benchmark_data_smithi031_454752_object28:head# 2024-04-25T09:15:57.544 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.577 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8941388a:::benchmark_data_smithi031_454752_object87:head# 2024-04-25T09:15:57.577 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.611 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8a4773c4:::benchmark_data_smithi031_454752_object103:head# 2024-04-25T09:15:57.611 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.644 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8abbf253:::benchmark_data_smithi031_454752_object97:head# 2024-04-25T09:15:57.644 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.677 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8adf42cb:::benchmark_data_smithi031_454752_object180:head# 2024-04-25T09:15:57.677 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.711 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8ae888df:::benchmark_data_smithi031_454752_object382:head# 2024-04-25T09:15:57.711 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.744 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8b291374:::benchmark_data_smithi031_454752_object273:head# 2024-04-25T09:15:57.745 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.778 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8b621052:::benchmark_data_smithi031_454752_object368:head# 2024-04-25T09:15:57.778 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.812 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8c90b5b1:::benchmark_data_smithi031_454752_object161:head# 2024-04-25T09:15:57.812 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.845 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8d29fb8d:::benchmark_data_smithi031_454752_object327:head# 2024-04-25T09:15:57.845 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.878 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8db4e958:::benchmark_data_smithi031_454752_object385:head# 2024-04-25T09:15:57.878 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.911 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8fa26b27:::benchmark_data_smithi031_454752_object256:head# 2024-04-25T09:15:57.912 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.945 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:90588a1a:::benchmark_data_smithi031_454752_object378:head# 2024-04-25T09:15:57.945 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.970 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:907e8918:::benchmark_data_smithi031_454752_object266:head# 2024-04-25T09:15:57.970 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:57.995 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9277f059:::benchmark_data_smithi031_454752_object2:head# 2024-04-25T09:15:57.995 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.029 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:946e6356:::benchmark_data_smithi031_454752_object338:head# 2024-04-25T09:15:58.029 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.062 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9586170f:::benchmark_data_smithi031_454752_object13:head# 2024-04-25T09:15:58.062 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.096 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:95c08ad2:::benchmark_data_smithi031_454752_object210:head# 2024-04-25T09:15:58.096 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.129 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:95cd8cdf:::benchmark_data_smithi031_454752_object383:head# 2024-04-25T09:15:58.129 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.162 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9609476c:::benchmark_data_smithi031_454752_object114:head# 2024-04-25T09:15:58.162 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.196 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:967bc283:::benchmark_data_smithi031_454752_object231:head# 2024-04-25T09:15:58.196 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.229 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96801ae8:::benchmark_data_smithi031_454752_object207:head# 2024-04-25T09:15:58.229 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.263 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96a964d2:::benchmark_data_smithi031_454752_object38:head# 2024-04-25T09:15:58.263 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.296 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96aa787d:::benchmark_data_smithi031_454752_object322:head# 2024-04-25T09:15:58.296 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.330 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96d63edc:::benchmark_data_smithi031_454752_object352:head# 2024-04-25T09:15:58.330 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.363 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:98e8b5bf:::benchmark_data_smithi031_454752_object216:head# 2024-04-25T09:15:58.363 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.397 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:99e3add0:::benchmark_data_smithi031_454752_object5:head# 2024-04-25T09:15:58.397 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.430 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a07c74e:::benchmark_data_smithi031_454752_object252:head# 2024-04-25T09:15:58.430 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.463 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a72fd5f:::benchmark_data_smithi031_454752_object297:head# 2024-04-25T09:15:58.463 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.497 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a9f4dbd:::benchmark_data_smithi031_454752_object321:head# 2024-04-25T09:15:58.497 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.530 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9b86cf51:::benchmark_data_smithi031_454752_object225:head# 2024-04-25T09:15:58.530 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.564 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9be87125:::benchmark_data_smithi031_454752_object116:head# 2024-04-25T09:15:58.564 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.597 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9bf0e6c9:::benchmark_data_smithi031_454752_object47:head# 2024-04-25T09:15:58.597 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.631 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9c836d36:::benchmark_data_smithi031_454752_object44:head# 2024-04-25T09:15:58.631 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.664 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9cb1d2e2:::benchmark_data_smithi031_454752_object362:head# 2024-04-25T09:15:58.664 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.698 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9cbe41c2:::benchmark_data_smithi031_454752_object374:head# 2024-04-25T09:15:58.698 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.723 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9ce31ece:::benchmark_data_smithi031_454752_object308:head# 2024-04-25T09:15:58.723 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.756 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9df3f789:::benchmark_data_smithi031_454752_object394:head# 2024-04-25T09:15:58.756 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.790 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9e952085:::benchmark_data_smithi031_454752_object154:head# 2024-04-25T09:15:58.790 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.873 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9f2a7ece:::benchmark_data_smithi031_454752_object251:head# 2024-04-25T09:15:58.873 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.907 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a114ad36:::benchmark_data_smithi031_454752_object70:head# 2024-04-25T09:15:58.907 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.932 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a419031b:::benchmark_data_smithi031_454752_object81:head# 2024-04-25T09:15:58.932 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.965 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a45f5752:::benchmark_data_smithi031_454752_object39:head# 2024-04-25T09:15:58.965 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:58.999 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a46e3cab:::benchmark_data_smithi031_454752_object193:head# 2024-04-25T09:15:58.999 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.049 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a46e59f0:::benchmark_data_smithi031_454752_object130:head# 2024-04-25T09:15:59.049 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.145 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a75374ab:::benchmark_data_smithi031_454752_object190:head# 2024-04-25T09:15:59.145 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.216 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8271a67:::benchmark_data_smithi031_454752_object33:head# 2024-04-25T09:15:59.217 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.258 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a853f81a:::benchmark_data_smithi031_454752_object379:head# 2024-04-25T09:15:59.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.300 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8c6bb46:::benchmark_data_smithi031_454752_object72:head# 2024-04-25T09:15:59.300 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.342 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a942e3ba:::benchmark_data_smithi031_454752_object74:head# 2024-04-25T09:15:59.342 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.383 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a98c4d7c:::benchmark_data_smithi031_454752_object37:head# 2024-04-25T09:15:59.384 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9e307d2:::benchmark_data_smithi031_454752_object318:head# 2024-04-25T09:15:59.425 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.467 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aab383a6:::benchmark_data_smithi031_454752_object112:head# 2024-04-25T09:15:59.467 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.509 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aaf58300:::benchmark_data_smithi031_454752_object208:head# 2024-04-25T09:15:59.509 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.551 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab0f7dc2:::benchmark_data_smithi031_454752_object119:head# 2024-04-25T09:15:59.551 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.584 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab1cf81e:::benchmark_data_smithi031_454752_object263:head# 2024-04-25T09:15:59.584 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.626 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab49749c:::benchmark_data_smithi031_454752_object359:head# 2024-04-25T09:15:59.626 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.668 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ac1d5d1f:::benchmark_data_smithi031_454752_object135:head# 2024-04-25T09:15:59.668 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.709 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ac7ea5bf:::benchmark_data_smithi031_454752_object217:head# 2024-04-25T09:15:59.710 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.760 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:acc3e569:::benchmark_data_smithi031_454752_object290:head# 2024-04-25T09:15:59.760 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.801 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ad398d0f:::benchmark_data_smithi031_454752_object125:head# 2024-04-25T09:15:59.802 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.843 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ae91e1ef:::benchmark_data_smithi031_454752_object115:head# 2024-04-25T09:15:59.843 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.877 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aea3dfa5:::benchmark_data_smithi031_454752_object388:head# 2024-04-25T09:15:59.877 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.919 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aec2dd45:::benchmark_data_smithi031_454752_object275:head# 2024-04-25T09:15:59.919 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:15:59.960 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af668e34:::benchmark_data_smithi031_454752_object302:head# 2024-04-25T09:15:59.960 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.027 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af9a9573:::benchmark_data_smithi031_454752_object58:head# 2024-04-25T09:16:00.027 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.060 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b089bb4f:::benchmark_data_smithi031_454752_object218:head# 2024-04-25T09:16:00.061 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.094 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b1674de8:::benchmark_data_smithi031_454752_object194:head# 2024-04-25T09:16:00.094 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.127 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b2930ee9:::benchmark_data_smithi031_454752_object73:head# 2024-04-25T09:16:00.128 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.169 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b329d82c:::benchmark_data_smithi031_454752_object200:head# 2024-04-25T09:16:00.169 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.211 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b3c75192:::benchmark_data_smithi031_454752_object325:head# 2024-04-25T09:16:00.211 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.253 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b45e9911:::benchmark_data_smithi031_454752_object53:head# 2024-04-25T09:16:00.253 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.295 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b4f7e75f:::benchmark_data_smithi031_454752_object296:head# 2024-04-25T09:16:00.295 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.336 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b51eab52:::benchmark_data_smithi031_454752_object393:head# 2024-04-25T09:16:00.336 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.378 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b54079d2:::benchmark_data_smithi031_454752_object319:head# 2024-04-25T09:16:00.378 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.420 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b5427b66:::benchmark_data_smithi031_454752_object113:head# 2024-04-25T09:16:00.420 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.453 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b59f1134:::benchmark_data_smithi031_454752_object301:head# 2024-04-25T09:16:00.453 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.487 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b66ddae8:::benchmark_data_smithi031_454752_object206:head# 2024-04-25T09:16:00.487 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.520 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b6a31cad:::benchmark_data_smithi031_454752_object27:head# 2024-04-25T09:16:00.520 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.554 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b7073e4b:::benchmark_data_smithi031_454752_object126:head# 2024-04-25T09:16:00.554 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.587 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b83f72b1:::benchmark_data_smithi031_454752_object162:head# 2024-04-25T09:16:00.587 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.621 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:16:00.621 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.654 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b94a5f10:::benchmark_data_smithi031_454752_object348:head# 2024-04-25T09:16:00.654 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.687 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b9a0d9e0:::benchmark_data_smithi031_454752_object342:head# 2024-04-25T09:16:00.688 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.721 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb052c00:::benchmark_data_smithi031_454752_object88:head# 2024-04-25T09:16:00.721 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.754 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb48e6d2:::benchmark_data_smithi031_454752_object241:head# 2024-04-25T09:16:00.754 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.779 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb57b519:::benchmark_data_smithi031_454752_object50:head# 2024-04-25T09:16:00.779 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.838 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb6af9e7:::benchmark_data_smithi031_454752_object373:head# 2024-04-25T09:16:00.838 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.888 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bca0f028:::benchmark_data_smithi031_454752_object20:head# 2024-04-25T09:16:00.889 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.922 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd00cd0f:::benchmark_data_smithi031_454752_object24:head# 2024-04-25T09:16:00.922 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.955 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd3a993f:::benchmark_data_smithi031_454752_object255:head# 2024-04-25T09:16:00.955 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:00.988 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bdd858f4:::benchmark_data_smithi031_454752_object228:head# 2024-04-25T09:16:00.989 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.022 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bfa77210:::benchmark_data_smithi031_454752_object67:head# 2024-04-25T09:16:01.022 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.081 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0619f0c:::benchmark_data_smithi031_454752_object247:head# 2024-04-25T09:16:01.081 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.136 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0b886c3:::benchmark_data_smithi031_454752_object25:head# 2024-04-25T09:16:01.137 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.206 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c229294b:::benchmark_data_smithi031_454752_object127:head# 2024-04-25T09:16:01.206 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c2f8ad7b:::benchmark_data_smithi031_454752_object104:head# 2024-04-25T09:16:01.240 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.273 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c4fe132d:::benchmark_data_smithi031_454752_object152:head# 2024-04-25T09:16:01.274 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.307 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c5dc4341:::benchmark_data_smithi031_454752_object166:head# 2024-04-25T09:16:01.307 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.340 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c6367955:::benchmark_data_smithi031_454752_object144:head# 2024-04-25T09:16:01.341 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.374 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c664156b:::benchmark_data_smithi031_454752_object238:head# 2024-04-25T09:16:01.374 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.407 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c78167ae:::benchmark_data_smithi031_454752_object85:head# 2024-04-25T09:16:01.407 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.441 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c79d3970:::benchmark_data_smithi031_454752_object221:head# 2024-04-25T09:16:01.441 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.474 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c8eb0827:::benchmark_data_smithi031_454752_object298:head# 2024-04-25T09:16:01.474 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.507 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca1d51a2:::benchmark_data_smithi031_454752_object280:head# 2024-04-25T09:16:01.508 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.541 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cab67049:::benchmark_data_smithi031_454752_object91:head# 2024-04-25T09:16:01.541 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.574 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cad688fc:::benchmark_data_smithi031_454752_object351:head# 2024-04-25T09:16:01.575 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.608 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:caf98509:::benchmark_data_smithi031_454752_object196:head# 2024-04-25T09:16:01.608 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.641 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc58c178:::benchmark_data_smithi031_454752_object32:head# 2024-04-25T09:16:01.641 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.675 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc9c514e:::benchmark_data_smithi031_454752_object65:head# 2024-04-25T09:16:01.675 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.708 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cca29ffb:::benchmark_data_smithi031_454752_object80:head# 2024-04-25T09:16:01.708 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.742 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ccbd2f1f:::benchmark_data_smithi031_454752_object184:head# 2024-04-25T09:16:01.742 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.775 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cd8693fa:::benchmark_data_smithi031_454752_object398:head# 2024-04-25T09:16:01.775 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.808 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ceb870b0:::benchmark_data_smithi031_454752_object222:head# 2024-04-25T09:16:01.809 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d093eb2e:::benchmark_data_smithi031_454752_object311:head# 2024-04-25T09:16:01.842 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.875 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d231d661:::benchmark_data_smithi031_454752_object177:head# 2024-04-25T09:16:01.876 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.909 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d2ae4d7f:::benchmark_data_smithi031_454752_object51:head# 2024-04-25T09:16:01.909 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.942 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d35c9cea:::benchmark_data_smithi031_454752_object317:head# 2024-04-25T09:16:01.942 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:01.984 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d4438560:::benchmark_data_smithi031_454752_object143:head# 2024-04-25T09:16:01.984 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.018 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d452fc99:::benchmark_data_smithi031_454752_object45:head# 2024-04-25T09:16:02.018 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d6574e4a:::benchmark_data_smithi031_454752_object60:head# 2024-04-25T09:16:02.068 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.110 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d73435c2:::benchmark_data_smithi031_454752_object118:head# 2024-04-25T09:16:02.110 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.151 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75560e8:::benchmark_data_smithi031_454752_object128:head# 2024-04-25T09:16:02.152 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.193 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75a04bb:::benchmark_data_smithi031_454752_object98:head# 2024-04-25T09:16:02.193 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.235 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75b7183:::benchmark_data_smithi031_454752_object230:head# 2024-04-25T09:16:02.235 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.277 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d7bf5e5b:::benchmark_data_smithi031_454752_object239:head# 2024-04-25T09:16:02.277 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.319 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d824c896:::benchmark_data_smithi031_454752_object286:head# 2024-04-25T09:16:02.319 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.360 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da211111:::benchmark_data_smithi031_454752_object159:head# 2024-04-25T09:16:02.361 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.402 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da25a4ff:::benchmark_data_smithi031_454752_object75:head# 2024-04-25T09:16:02.402 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.444 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:db18fece:::benchmark_data_smithi031_454752_object250:head# 2024-04-25T09:16:02.444 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.486 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dc4325f0:::benchmark_data_smithi031_454752_object131:head# 2024-04-25T09:16:02.486 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.528 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dedcc3de:::benchmark_data_smithi031_454752_object148:head# 2024-04-25T09:16:02.528 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.569 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:deec6ff8:::benchmark_data_smithi031_454752_object224:head# 2024-04-25T09:16:02.569 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.611 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e0881e69:::benchmark_data_smithi031_454752_object293:head# 2024-04-25T09:16:02.611 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.653 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e1341170:::benchmark_data_smithi031_454752_object132:head# 2024-04-25T09:16:02.653 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.695 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e17ed360:::benchmark_data_smithi031_454752_object140:head# 2024-04-25T09:16:02.695 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.736 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3be378c:::benchmark_data_smithi031_454752_object188:head# 2024-04-25T09:16:02.737 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.778 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3c37152:::benchmark_data_smithi031_454752_object155:head# 2024-04-25T09:16:02.778 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.845 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e431150e:::benchmark_data_smithi031_454752_object309:head# 2024-04-25T09:16:02.845 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.904 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e44b8a2c:::benchmark_data_smithi031_454752_object146:head# 2024-04-25T09:16:02.904 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.946 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e4a3dd60:::benchmark_data_smithi031_454752_object341:head# 2024-04-25T09:16:02.946 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:02.987 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e53228e8:::benchmark_data_smithi031_454752_object364:head# 2024-04-25T09:16:02.987 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.029 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e53430a0:::benchmark_data_smithi031_454752_object336:head# 2024-04-25T09:16:03.029 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.088 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e5497360:::benchmark_data_smithi031_454752_object141:head# 2024-04-25T09:16:03.088 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.138 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e5ae028f:::benchmark_data_smithi031_454752_object254:head# 2024-04-25T09:16:03.138 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.188 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e72dd955:::benchmark_data_smithi031_454752_object105:head# 2024-04-25T09:16:03.188 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.222 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e75527e1:::benchmark_data_smithi031_454752_object365:head# 2024-04-25T09:16:03.222 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.255 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e8e4cd11:::benchmark_data_smithi031_454752_object171:head# 2024-04-25T09:16:03.255 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.288 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e94a0572:::benchmark_data_smithi031_454752_object6:head# 2024-04-25T09:16:03.289 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.314 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e991363a:::benchmark_data_smithi031_454752_object52:head# 2024-04-25T09:16:03.314 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.347 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ea70172b:::benchmark_data_smithi031_454752_object191:head# 2024-04-25T09:16:03.347 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.381 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:eaf3d89c:::benchmark_data_smithi031_454752_object358:head# 2024-04-25T09:16:03.381 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.414 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ec070df6:::benchmark_data_smithi031_454752_object234:head# 2024-04-25T09:16:03.414 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.447 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ec82b70c:::benchmark_data_smithi031_454752_object246:head# 2024-04-25T09:16:03.448 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.481 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ed17f3a5:::benchmark_data_smithi031_454752_object389:head# 2024-04-25T09:16:03.481 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.514 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:edd73bf9:::benchmark_data_smithi031_454752_object95:head# 2024-04-25T09:16:03.515 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.548 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ee0f64f0:::benchmark_data_smithi031_454752_object49:head# 2024-04-25T09:16:03.548 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.581 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f054ad11:::benchmark_data_smithi031_454752_object145:head# 2024-04-25T09:16:03.581 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.615 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f102181e:::benchmark_data_smithi031_454752_object288:head# 2024-04-25T09:16:03.615 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.648 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f21d79de:::benchmark_data_smithi031_454752_object66:head# 2024-04-25T09:16:03.648 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.682 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f2b747ee:::benchmark_data_smithi031_454752_object314:head# 2024-04-25T09:16:03.682 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.715 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f34b8ad2:::benchmark_data_smithi031_454752_object240:head# 2024-04-25T09:16:03.715 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.749 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4863048:::benchmark_data_smithi031_454752_object349:head# 2024-04-25T09:16:03.749 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.783 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4ab8147:::benchmark_data_smithi031_454752_object337:head# 2024-04-25T09:16:03.783 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f547a4d2:::benchmark_data_smithi031_454752_object369:head# 2024-04-25T09:16:03.845 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.882 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6236fee:::benchmark_data_smithi031_454752_object289:head# 2024-04-25T09:16:03.882 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f635f136:::benchmark_data_smithi031_454752_object23:head# 2024-04-25T09:16:03.916 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.949 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6e629fe:::benchmark_data_smithi031_454752_object57:head# 2024-04-25T09:16:03.949 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:03.983 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f70fb177:::benchmark_data_smithi031_454752_object265:head# 2024-04-25T09:16:03.983 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.016 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f77401b8:::benchmark_data_smithi031_454752_object71:head# 2024-04-25T09:16:04.016 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.049 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f870ff0e:::benchmark_data_smithi031_454752_object312:head# 2024-04-25T09:16:04.050 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.083 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:faefa441:::benchmark_data_smithi031_454752_object269:head# 2024-04-25T09:16:04.083 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.116 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb1204b7:::benchmark_data_smithi031_454752_object331:head# 2024-04-25T09:16:04.117 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.150 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb3355e2:::benchmark_data_smithi031_454752_object363:head# 2024-04-25T09:16:04.150 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb5c543a:::benchmark_data_smithi031_454752_object329:head# 2024-04-25T09:16:04.183 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.217 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fcf4159f:::benchmark_data_smithi031_454752_object185:head# 2024-04-25T09:16:04.217 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.250 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fe7c790a:::benchmark_data_smithi031_454752_object16:head# 2024-04-25T09:16:04.250 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.284 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ffeaaa0c:::benchmark_data_smithi031_454752_object139:head# 2024-04-25T09:16:04.284 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:04.317 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 10 info 1.1( v 10'189 (0'0,10'189] local-lis/les=7/8 n=189 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:16:04.359 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T09:16:04.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:16:05.992 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e21 pg_nums {1={7=2,20=1}} deleted_pools ) 2024-04-25T09:16:05.992 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T09:16:05.993 INFO:tasks.workunit.client.0.smithi031.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:16:05.993 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0005830b:::benchmark_data_smithi031_454752_object169:head# 2024-04-25T09:16:05.993 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.063 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0007ff59:::benchmark_data_smithi031_454752_object384:head# 2024-04-25T09:16:06.063 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.096 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00530cad:::benchmark_data_smithi031_454752_object366:head# 2024-04-25T09:16:06.096 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.129 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:007182ad:::benchmark_data_smithi031_454752_object150:head# 2024-04-25T09:16:06.130 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00808219:::benchmark_data_smithi031_454752_object10:head# 2024-04-25T09:16:06.163 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.196 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00ccfbba:::benchmark_data_smithi031_454752_object355:head# 2024-04-25T09:16:06.197 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.230 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00d7eda8:::benchmark_data_smithi031_454752_object357:head# 2024-04-25T09:16:06.230 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.263 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:01faf234:::benchmark_data_smithi031_454752_object272:head# 2024-04-25T09:16:06.263 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.297 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:021d5fe5:::benchmark_data_smithi031_454752_object109:head# 2024-04-25T09:16:06.297 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.322 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:02a87b95:::benchmark_data_smithi031_454752_object26:head# 2024-04-25T09:16:06.322 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.347 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03119e67:::benchmark_data_smithi031_454752_object120:head# 2024-04-25T09:16:06.347 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.372 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03a19a01:::benchmark_data_smithi031_454752_object375:head# 2024-04-25T09:16:06.372 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.398 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03d8d0d6:::benchmark_data_smithi031_454752_object307:head# 2024-04-25T09:16:06.399 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.422 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0407316c:::benchmark_data_smithi031_454752_object248:head# 2024-04-25T09:16:06.422 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.447 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:069ac774:::benchmark_data_smithi031_454752_object270:head# 2024-04-25T09:16:06.448 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.472 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:076ac3b3:::benchmark_data_smithi031_454752_object168:head# 2024-04-25T09:16:06.473 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.498 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:078ca8f4:::benchmark_data_smithi031_454752_object271:head# 2024-04-25T09:16:06.498 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.531 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:089737ee:::benchmark_data_smithi031_454752_object262:head# 2024-04-25T09:16:06.531 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.564 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0954f670:::benchmark_data_smithi031_454752_object223:head# 2024-04-25T09:16:06.565 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.598 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:09d319a2:::benchmark_data_smithi031_454752_object283:head# 2024-04-25T09:16:06.598 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a33ee0a:::benchmark_data_smithi031_454752_object107:head# 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.665 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a588e5f:::benchmark_data_smithi031_454752_object305:head# 2024-04-25T09:16:06.665 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.698 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ae0c865:::benchmark_data_smithi031_454752_object41:head# 2024-04-25T09:16:06.698 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.732 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0c38e0a3:::benchmark_data_smithi031_454752_object387:head# 2024-04-25T09:16:06.732 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.765 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0c4decad:::benchmark_data_smithi031_454752_object151:head# 2024-04-25T09:16:06.765 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.811 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0cc7bcb7:::benchmark_data_smithi031_454752_object333:head# 2024-04-25T09:16:06.811 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.859 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ce7b7bc:::benchmark_data_smithi031_454752_object353:head# 2024-04-25T09:16:06.989 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.990 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0d6e3436:::benchmark_data_smithi031_454752_object205:head# 2024-04-25T09:16:06.991 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.991 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0dd158a0:::benchmark_data_smithi031_454752_object76:head# 2024-04-25T09:16:06.991 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:06.991 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0df905cb:::benchmark_data_smithi031_454752_object181:head# 2024-04-25T09:16:06.991 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.008 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0f047668:::benchmark_data_smithi031_454752_object12:head# 2024-04-25T09:16:07.008 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.075 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:100b6e67:::benchmark_data_smithi031_454752_object122:head# 2024-04-25T09:16:07.075 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.142 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:105e9b29:::benchmark_data_smithi031_454752_object295:head# 2024-04-25T09:16:07.142 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.175 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:10ca342c:::benchmark_data_smithi031_454752_object203:head# 2024-04-25T09:16:07.176 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.209 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:11985b73:::benchmark_data_smithi031_454752_object294:head# 2024-04-25T09:16:07.209 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.243 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1215ebb0:::benchmark_data_smithi031_454752_object96:head# 2024-04-25T09:16:07.243 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.284 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:123f25d6:::benchmark_data_smithi031_454752_object339:head# 2024-04-25T09:16:07.284 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.318 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:12648b62:::benchmark_data_smithi031_454752_object235:head# 2024-04-25T09:16:07.318 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.351 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:13b5933b:::benchmark_data_smithi031_454752_object258:head# 2024-04-25T09:16:07.351 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.385 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:14a4f03a:::benchmark_data_smithi031_454752_object90:head# 2024-04-25T09:16:07.385 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.418 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:152756a2:::benchmark_data_smithi031_454752_object78:head# 2024-04-25T09:16:07.418 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.452 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17685eef:::benchmark_data_smithi031_454752_object35:head# 2024-04-25T09:16:07.452 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.485 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17d78309:::benchmark_data_smithi031_454752_object197:head# 2024-04-25T09:16:07.485 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.518 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:186323b7:::benchmark_data_smithi031_454752_object264:head# 2024-04-25T09:16:07.519 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.552 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18a96ee0:::benchmark_data_smithi031_454752_object340:head# 2024-04-25T09:16:07.552 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.585 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18abc847:::benchmark_data_smithi031_454752_object377:head# 2024-04-25T09:16:07.585 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.619 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:190ad911:::benchmark_data_smithi031_454752_object158:head# 2024-04-25T09:16:07.619 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:192b4b6c:::benchmark_data_smithi031_454752_object249:head# 2024-04-25T09:16:07.652 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.719 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1a2dbf6a:::benchmark_data_smithi031_454752_object284:head# 2024-04-25T09:16:07.719 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.752 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1bace3cc:::benchmark_data_smithi031_454752_object136:head# 2024-04-25T09:16:07.753 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.786 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1f6172c3:::benchmark_data_smithi031_454752_object315:head# 2024-04-25T09:16:07.786 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.819 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20988891:::benchmark_data_smithi031_454752_object0:head# 2024-04-25T09:16:07.819 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.853 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20c89311:::benchmark_data_smithi031_454752_object170:head# 2024-04-25T09:16:07.853 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.886 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20f96ee6:::benchmark_data_smithi031_454752_object30:head# 2024-04-25T09:16:07.886 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.920 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:210b81be:::benchmark_data_smithi031_454752_object396:head# 2024-04-25T09:16:07.920 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.953 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:222bd3b2:::benchmark_data_smithi031_454752_object244:head# 2024-04-25T09:16:07.953 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:07.986 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:23ea9d14:::benchmark_data_smithi031_454752_object134:head# 2024-04-25T09:16:07.987 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.021 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2564bee6:::benchmark_data_smithi031_454752_object68:head# 2024-04-25T09:16:08.021 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.053 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:26e5d41e:::benchmark_data_smithi031_454752_object261:head# 2024-04-25T09:16:08.054 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.087 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2765dc2c:::benchmark_data_smithi031_454752_object147:head# 2024-04-25T09:16:08.087 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.120 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:286a6080:::benchmark_data_smithi031_454752_object64:head# 2024-04-25T09:16:08.120 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.154 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2899a918:::benchmark_data_smithi031_454752_object267:head# 2024-04-25T09:16:08.154 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.192 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:299c9ca7:::benchmark_data_smithi031_454752_object21:head# 2024-04-25T09:16:08.192 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.281 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:29db69da:::benchmark_data_smithi031_454752_object236:head# 2024-04-25T09:16:08.281 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.355 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2aa178ac:::benchmark_data_smithi031_454752_object59:head# 2024-04-25T09:16:08.355 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.436 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ac07948:::benchmark_data_smithi031_454752_object7:head# 2024-04-25T09:16:08.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.520 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2af7345c:::benchmark_data_smithi031_454752_object202:head# 2024-04-25T09:16:08.520 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.573 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2c7cae34:::benchmark_data_smithi031_454752_object300:head# 2024-04-25T09:16:08.573 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.606 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2d714369:::benchmark_data_smithi031_454752_object292:head# 2024-04-25T09:16:08.606 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.639 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2da27532:::benchmark_data_smithi031_454752_object1:head# 2024-04-25T09:16:08.639 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.664 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2dee9738:::benchmark_data_smithi031_454752_object400:head# 2024-04-25T09:16:08.665 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.689 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e61c031:::benchmark_data_smithi031_454752_object174:head# 2024-04-25T09:16:08.690 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.714 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ee4a5f6:::benchmark_data_smithi031_454752_object94:head# 2024-04-25T09:16:08.715 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.740 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2f1e460a:::benchmark_data_smithi031_454752_object106:head# 2024-04-25T09:16:08.740 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.765 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2fc8a11e:::benchmark_data_smithi031_454752_object260:head# 2024-04-25T09:16:08.765 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.790 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:30797560:::benchmark_data_smithi031_454752_object142:head# 2024-04-25T09:16:08.790 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.875 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3105618d:::benchmark_data_smithi031_454752_object46:head# 2024-04-25T09:16:08.875 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.916 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31a53b00:::benchmark_data_smithi031_454752_object14:head# 2024-04-25T09:16:08.916 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.949 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31b8e0ab:::benchmark_data_smithi031_454752_object326:head# 2024-04-25T09:16:08.949 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:08.982 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31c961a5:::benchmark_data_smithi031_454752_object11:head# 2024-04-25T09:16:08.983 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.016 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31f4feb6:::benchmark_data_smithi031_454752_object54:head# 2024-04-25T09:16:09.016 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.083 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:332a7b27:::benchmark_data_smithi031_454752_object257:head# 2024-04-25T09:16:09.083 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.150 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:33cf1783:::benchmark_data_smithi031_454752_object83:head# 2024-04-25T09:16:09.150 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.183 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:33e69519:::benchmark_data_smithi031_454752_object334:head# 2024-04-25T09:16:09.184 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.217 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:343b464c:::benchmark_data_smithi031_454752_object229:head# 2024-04-25T09:16:09.217 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.250 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:353bc8d6:::benchmark_data_smithi031_454752_object306:head# 2024-04-25T09:16:09.251 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.284 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:357adf2e:::benchmark_data_smithi031_454752_object313:head# 2024-04-25T09:16:09.284 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.326 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3626a3d2:::benchmark_data_smithi031_454752_object242:head# 2024-04-25T09:16:09.326 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.359 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:366a3868:::benchmark_data_smithi031_454752_object344:head# 2024-04-25T09:16:09.359 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.393 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:36e1e4b7:::benchmark_data_smithi031_454752_object330:head# 2024-04-25T09:16:09.393 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.426 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3747f9e2:::benchmark_data_smithi031_454752_object361:head# 2024-04-25T09:16:09.426 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.460 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:376c5fd1:::benchmark_data_smithi031_454752_object276:head# 2024-04-25T09:16:09.460 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.493 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:37ad6f44:::benchmark_data_smithi031_454752_object199:head# 2024-04-25T09:16:09.493 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.526 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3809a39d:::benchmark_data_smithi031_454752_object63:head# 2024-04-25T09:16:09.527 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.560 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:38962184:::benchmark_data_smithi031_454752_object345:head# 2024-04-25T09:16:09.560 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.593 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3947aa59:::benchmark_data_smithi031_454752_object29:head# 2024-04-25T09:16:09.594 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.627 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:39a62bf7:::benchmark_data_smithi031_454752_object346:head# 2024-04-25T09:16:09.627 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.660 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3a70fad2:::benchmark_data_smithi031_454752_object213:head# 2024-04-25T09:16:09.660 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.694 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3aaccaa2:::benchmark_data_smithi031_454752_object245:head# 2024-04-25T09:16:09.694 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.727 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b4d69f0:::benchmark_data_smithi031_454752_object133:head# 2024-04-25T09:16:09.727 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.761 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b5ed4ad:::benchmark_data_smithi031_454752_object153:head# 2024-04-25T09:16:09.761 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.794 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3bd7c293:::benchmark_data_smithi031_454752_object226:head# 2024-04-25T09:16:09.794 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.828 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3d9bbce6:::benchmark_data_smithi031_454752_object195:head# 2024-04-25T09:16:09.828 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.861 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3e1dceb1:::benchmark_data_smithi031_454752_object163:head# 2024-04-25T09:16:09.861 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.894 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f585de7:::benchmark_data_smithi031_454752_object370:head# 2024-04-25T09:16:09.895 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.928 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f8dddc0:::benchmark_data_smithi031_454752_object36:head# 2024-04-25T09:16:09.928 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.961 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:40e869df:::benchmark_data_smithi031_454752_object354:head# 2024-04-25T09:16:09.961 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:09.995 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:42f70b52:::benchmark_data_smithi031_454752_object391:head# 2024-04-25T09:16:09.995 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.028 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:44648b7f:::benchmark_data_smithi031_454752_object15:head# 2024-04-25T09:16:10.028 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.062 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:45c0069e:::benchmark_data_smithi031_454752_object93:head# 2024-04-25T09:16:10.062 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.095 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:46cce1fb:::benchmark_data_smithi031_454752_object259:head# 2024-04-25T09:16:10.095 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.128 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48372743:::benchmark_data_smithi031_454752_object92:head# 2024-04-25T09:16:10.129 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.162 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:487de23f:::benchmark_data_smithi031_454752_object278:head# 2024-04-25T09:16:10.162 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.195 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48933b4f:::benchmark_data_smithi031_454752_object219:head# 2024-04-25T09:16:10.195 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.229 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48b43814:::benchmark_data_smithi031_454752_object22:head# 2024-04-25T09:16:10.229 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.262 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:49675aeb:::benchmark_data_smithi031_454752_object99:head# 2024-04-25T09:16:10.262 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.296 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a2d5412:::benchmark_data_smithi031_454752_object324:head# 2024-04-25T09:16:10.296 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.329 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a6e0d41:::benchmark_data_smithi031_454752_object268:head# 2024-04-25T09:16:10.329 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.363 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a6e92f8:::benchmark_data_smithi031_454752_object198:head# 2024-04-25T09:16:10.363 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.404 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a95c670:::benchmark_data_smithi031_454752_object220:head# 2024-04-25T09:16:10.405 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.438 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a9a32d5:::benchmark_data_smithi031_454752_object56:head# 2024-04-25T09:16:10.438 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.471 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4aa2f1b1:::benchmark_data_smithi031_454752_object160:head# 2024-04-25T09:16:10.472 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.530 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4ac6a300:::benchmark_data_smithi031_454752_object209:head# 2024-04-25T09:16:10.530 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.563 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b065b88:::benchmark_data_smithi031_454752_object34:head# 2024-04-25T09:16:10.563 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.597 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b3ae5fa:::benchmark_data_smithi031_454752_object399:head# 2024-04-25T09:16:10.597 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.630 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b491b4d:::benchmark_data_smithi031_454752_object86:head# 2024-04-25T09:16:10.630 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.663 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4c281e67:::benchmark_data_smithi031_454752_object123:head# 2024-04-25T09:16:10.664 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd1274c:::benchmark_data_smithi031_454752_object77:head# 2024-04-25T09:16:10.699 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.742 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd91fcf:::benchmark_data_smithi031_454752_object227:head# 2024-04-25T09:16:10.742 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.816 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd9e311:::benchmark_data_smithi031_454752_object165:head# 2024-04-25T09:16:10.816 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.864 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4e704b66:::benchmark_data_smithi031_454752_object110:head# 2024-04-25T09:16:10.865 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.915 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4f141352:::benchmark_data_smithi031_454752_object390:head# 2024-04-25T09:16:10.915 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.948 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4f67372e:::benchmark_data_smithi031_454752_object310:head# 2024-04-25T09:16:10.948 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:10.981 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5055bf66:::benchmark_data_smithi031_454752_object111:head# 2024-04-25T09:16:10.981 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.025 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5079fdc4:::benchmark_data_smithi031_454752_object102:head# 2024-04-25T09:16:11.025 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.102 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51592368:::benchmark_data_smithi031_454752_object129:head# 2024-04-25T09:16:11.102 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.166 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51c35831:::benchmark_data_smithi031_454752_object277:head# 2024-04-25T09:16:11.166 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.191 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:525ef634:::benchmark_data_smithi031_454752_object303:head# 2024-04-25T09:16:11.191 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.216 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:52f1a683:::benchmark_data_smithi031_454752_object232:head# 2024-04-25T09:16:11.216 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.241 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:549943cc:::benchmark_data_smithi031_454752_object137:head# 2024-04-25T09:16:11.241 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.266 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:54a7cad2:::benchmark_data_smithi031_454752_object243:head# 2024-04-25T09:16:11.266 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.310 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56c9c8a3:::benchmark_data_smithi031_454752_object386:head# 2024-04-25T09:16:11.311 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.333 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56d22a80:::benchmark_data_smithi031_454752_object61:head# 2024-04-25T09:16:11.333 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.366 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:576c3dbd:::benchmark_data_smithi031_454752_object320:head# 2024-04-25T09:16:11.366 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.400 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:580c5a46:::benchmark_data_smithi031_454752_object62:head# 2024-04-25T09:16:11.400 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.433 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:58815a7c:::benchmark_data_smithi031_454752_object392:head# 2024-04-25T09:16:11.434 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.467 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:58e6f983:::benchmark_data_smithi031_454752_object233:head# 2024-04-25T09:16:11.467 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.525 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:59c109e7:::benchmark_data_smithi031_454752_object372:head# 2024-04-25T09:16:11.525 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.567 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a7c4e16:::benchmark_data_smithi031_454752_object48:head# 2024-04-25T09:16:11.568 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.601 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a943612:::benchmark_data_smithi031_454752_object40:head# 2024-04-25T09:16:11.601 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.634 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5aed2644:::benchmark_data_smithi031_454752_object395:head# 2024-04-25T09:16:11.634 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.668 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5c81318e:::benchmark_data_smithi031_454752_object18:head# 2024-04-25T09:16:11.668 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.701 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5d21802c:::benchmark_data_smithi031_454752_object201:head# 2024-04-25T09:16:11.701 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.735 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5d50b842:::benchmark_data_smithi031_454752_object282:head# 2024-04-25T09:16:11.735 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5de1e511:::benchmark_data_smithi031_454752_object173:head# 2024-04-25T09:16:11.768 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.802 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e1e382c:::benchmark_data_smithi031_454752_object189:head# 2024-04-25T09:16:11.802 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.835 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e3603a8:::benchmark_data_smithi031_454752_object356:head# 2024-04-25T09:16:11.835 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.869 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e9a3526:::benchmark_data_smithi031_454752_object274:head# 2024-04-25T09:16:11.869 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.902 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:60dee496:::benchmark_data_smithi031_454752_object287:head# 2024-04-25T09:16:11.902 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.935 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:610072df:::benchmark_data_smithi031_454752_object381:head# 2024-04-25T09:16:11.936 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:11.971 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:61757bc4:::benchmark_data_smithi031_454752_object100:head# 2024-04-25T09:16:11.971 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.002 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:61eb06a1:::benchmark_data_smithi031_454752_object42:head# 2024-04-25T09:16:12.003 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.036 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:622a095a:::benchmark_data_smithi031_454752_object172:head# 2024-04-25T09:16:12.036 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.070 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:629696e0:::benchmark_data_smithi031_454752_object343:head# 2024-04-25T09:16:12.070 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.103 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:62dff125:::benchmark_data_smithi031_454752_object117:head# 2024-04-25T09:16:12.103 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.136 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:637e2a8b:::benchmark_data_smithi031_454752_object124:head# 2024-04-25T09:16:12.136 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.169 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:63a84c81:::benchmark_data_smithi031_454752_object82:head# 2024-04-25T09:16:12.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.203 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:64a1f0ad:::benchmark_data_smithi031_454752_object367:head# 2024-04-25T09:16:12.203 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.236 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:65a082ea:::benchmark_data_smithi031_454752_object316:head# 2024-04-25T09:16:12.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.270 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:66d0a0c5:::benchmark_data_smithi031_454752_object4:head# 2024-04-25T09:16:12.270 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.303 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6757b3f7:::benchmark_data_smithi031_454752_object347:head# 2024-04-25T09:16:12.303 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.337 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68851dbf:::benchmark_data_smithi031_454752_object279:head# 2024-04-25T09:16:12.337 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.370 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68869d9f:::benchmark_data_smithi031_454752_object8:head# 2024-04-25T09:16:12.370 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.404 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6922a669:::benchmark_data_smithi031_454752_object291:head# 2024-04-25T09:16:12.404 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:695ddfd2:::benchmark_data_smithi031_454752_object285:head# 2024-04-25T09:16:12.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.470 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:69f748df:::benchmark_data_smithi031_454752_object380:head# 2024-04-25T09:16:12.471 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.504 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6a4d4ed2:::benchmark_data_smithi031_454752_object212:head# 2024-04-25T09:16:12.504 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.537 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d7df3bd:::benchmark_data_smithi031_454752_object323:head# 2024-04-25T09:16:12.537 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.571 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d9bcab7:::benchmark_data_smithi031_454752_object299:head# 2024-04-25T09:16:12.571 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.604 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6e1dd161:::benchmark_data_smithi031_454752_object176:head# 2024-04-25T09:16:12.605 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.638 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ea26cda:::benchmark_data_smithi031_454752_object237:head# 2024-04-25T09:16:12.638 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.671 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:715f84d9:::benchmark_data_smithi031_454752_object31:head# 2024-04-25T09:16:12.671 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.704 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:718b1888:::benchmark_data_smithi031_454752_object3:head# 2024-04-25T09:16:12.705 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7201ba57:::benchmark_data_smithi031_454752_object332:head# 2024-04-25T09:16:12.738 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.771 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72916a4f:::benchmark_data_smithi031_454752_object43:head# 2024-04-25T09:16:12.772 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.816 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72e5e7bc:::benchmark_data_smithi031_454752_object350:head# 2024-04-25T09:16:12.816 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.863 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72f2ccd2:::benchmark_data_smithi031_454752_object211:head# 2024-04-25T09:16:12.864 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.914 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:737dcd92:::benchmark_data_smithi031_454752_object335:head# 2024-04-25T09:16:12.914 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.947 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:744eb9c4:::benchmark_data_smithi031_454752_object101:head# 2024-04-25T09:16:12.947 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:12.981 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75451e3a:::benchmark_data_smithi031_454752_object215:head# 2024-04-25T09:16:12.981 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.014 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75b87d24:::benchmark_data_smithi031_454752_object156:head# 2024-04-25T09:16:13.014 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.068 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76a44be7:::benchmark_data_smithi031_454752_object371:head# 2024-04-25T09:16:13.068 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76a5bace:::benchmark_data_smithi031_454752_object253:head# 2024-04-25T09:16:13.132 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.190 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:772eee67:::benchmark_data_smithi031_454752_object121:head# 2024-04-25T09:16:13.190 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.224 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:798c5654:::benchmark_data_smithi031_454752_object204:head# 2024-04-25T09:16:13.224 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.257 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7995db24:::benchmark_data_smithi031_454752_object157:head# 2024-04-25T09:16:13.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.290 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79abc611:::benchmark_data_smithi031_454752_object164:head# 2024-04-25T09:16:13.291 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.324 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79d1490c:::benchmark_data_smithi031_454752_object138:head# 2024-04-25T09:16:13.324 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.357 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7b6edc17:::benchmark_data_smithi031_454752_object214:head# 2024-04-25T09:16:13.358 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7b93ca23:::benchmark_data_smithi031_454752_object55:head# 2024-04-25T09:16:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.433 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bd5320a:::benchmark_data_smithi031_454752_object328:head# 2024-04-25T09:16:13.433 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.466 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bde86cb:::benchmark_data_smithi031_454752_object182:head# 2024-04-25T09:16:13.466 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.500 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bfed9be:::benchmark_data_smithi031_454752_object397:head# 2024-04-25T09:16:13.500 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.533 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7c0b3015:::benchmark_data_smithi031_454752_object108:head# 2024-04-25T09:16:13.533 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.566 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7cb420f8:::benchmark_data_smithi031_454752_object89:head# 2024-04-25T09:16:13.567 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.600 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7d588f87:::benchmark_data_smithi031_454752_object376:head# 2024-04-25T09:16:13.600 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.633 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dbcb22e:::benchmark_data_smithi031_454752_object17:head# 2024-04-25T09:16:13.633 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.667 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dc6b3de:::benchmark_data_smithi031_454752_object187:head# 2024-04-25T09:16:13.667 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.692 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dec0b89:::benchmark_data_smithi031_454752_object149:head# 2024-04-25T09:16:13.692 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.717 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dfe5ddd:::benchmark_data_smithi031_454752_object79:head# 2024-04-25T09:16:13.717 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.751 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7ea5a1e2:::benchmark_data_smithi031_454752_object360:head# 2024-04-25T09:16:13.751 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.776 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7eeb5bde:::benchmark_data_smithi031_454752_object186:head# 2024-04-25T09:16:13.776 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.801 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7fb12f34:::benchmark_data_smithi031_454752_object69:head# 2024-04-25T09:16:13.801 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:16:13.834 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 10 info 1.0( v 10'213 (0'0,10'213] local-lis/les=7/8 n=213 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:16:13.868 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T09:16:14.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:16:14.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:16:14.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:16:14.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:16:14.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:16:14.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:16:14.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:16:14.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:16:14.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:16:14.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:14.411 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:14.412 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:16:14.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:16:14.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:16:14.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:16:14.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:16:14.414 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:16:14.415 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:16:14.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:16:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:16:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:16:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:16:14.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:16:14.450 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:16:14.461+0000 7f97640e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:14.452 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:16:14.465+0000 7f97640e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:14.454 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:16:14.465+0000 7f97640e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:14.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:16:14.916 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:16:14.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:14.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:16:15.031 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:16:15.040+0000 7f97640e87c0 -1 Falling back to public interface 2024-04-25T09:16:15.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:16.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:16.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:16.357 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:16:16.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:16:16.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:16.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:16:16.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:17.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:17.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:17.786 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:16:17.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:16:17.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:17.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:16:17.855 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:16:17.864+0000 7f97640e87c0 -1 osd.0 21 log_to_monitors true 2024-04-25T09:16:18.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:19.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:19.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:19.217 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:16:19.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:16:19.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:19.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:16:19.684 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [13,21) [v2:127.0.0.1:6802/3601708316,v1:127.0.0.1:6803/3601708316] [v2:127.0.0.1:6804/3601708316,v1:127.0.0.1:6805/3601708316] exists,up d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:16:19.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:16:19.686 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:16:19.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:16:19.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:16:19.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:16:19.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:16:19.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:16:19.790 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:16:20.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:16:20.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:16:20.197 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:20.198 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:16:20.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215107 2024-04-25T09:16:20.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215107 2024-04-25T09:16:20.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215107' 2024-04-25T09:16:20.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:20.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215107 2024-04-25T09:16:20.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:20.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:16:20.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215107 2024-04-25T09:16:20.365 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:20.365 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215107 2024-04-25T09:16:20.366 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 103079215107 2024-04-25T09:16:20.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215107' 2024-04-25T09:16:20.366 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:16:20.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 103079215107 2024-04-25T09:16:20.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:16:21.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:16:21.839 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:16:22.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215107 -lt 103079215107 2024-04-25T09:16:22.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:16:22.316 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:22.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:22.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:16:22.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:16:22.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:16:22.887 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:16:22.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:16:22.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:16:22.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:16:22.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:16:23.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:16:23.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:16:23.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:23.311 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:23.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:16:23.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:16:23.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:16:23.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-04-25T09:16:23.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:23.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:24.574 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:24.594 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:25.385 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:25.403 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:25.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:26.130 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:26.991 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:27.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:27.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:27.729 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:27.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:28.583 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:28.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:28.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:29.386 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:29.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:30.148 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:30.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:30.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:30.909 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:30.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:31.654 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:31.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:31.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:32.415 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:32.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:33.194 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:33.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:33.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:33.947 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:33.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:34.683 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:34.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:34.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:35.428 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:35.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:36.198 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:36.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:36.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:36.951 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:36.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:37.697 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:37.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:37.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:38.558 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:38.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:39.453 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:39.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:39.475 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:40.155 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:40.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:41.009 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:41.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:41.034 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:41.754 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:41.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:42.440 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:42.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:42.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:43.160 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:43.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:43.904 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:43.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:43.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:44.649 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:44.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:45.319 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:45.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:46.091 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:46.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:46.858 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:46.879 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:46.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:47.603 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:47.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:48.431 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:48.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:48.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:49.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:49.854 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:49.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:49.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:50.640 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:50.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:51.469 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:51.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:51.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:52.238 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:52.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:52.999 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:53.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:53.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:53.779 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:53.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:54.521 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:54.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:54.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:55.392 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:55.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:56.086 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:56.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:56.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:56.906 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:56.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:57.675 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:57.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:57.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:16:58.436 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:16:58.461 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:16:59.198 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:16:59.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:16:59.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:00.052 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:00.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:00.737 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:00.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:00.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:01.599 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:01.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:02.419 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:02.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:02.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:03.155 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:03.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:03.992 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:04.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:04.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:04.694 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:04.716 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:05.413 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:05.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:05.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:06.142 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:06.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:07.037 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:07.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:07.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:07.749 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:07.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:08.729 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:08.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:08.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:09.591 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:09.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:10.351 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:10.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:10.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:11.095 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:11.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:11.832 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:11.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:11.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:12.594 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:12.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:13.405 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:13.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:13.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:14.150 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:14.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:14.945 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:14.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:14.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:15.731 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:15.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:16.408 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:16.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:16.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:17.195 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:17.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:17.914 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:17.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:17.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:18.701 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:18.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:19.471 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:19.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:19.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:20.324 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:20.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:21.077 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:21.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:21.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:21.813 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:21.837 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:22.717 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:22.744 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:22.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:23.478 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:23.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:24.340 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:24.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:24.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:25.151 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:25.173 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:25.845 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:25.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:25.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:26.589 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:26.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:27.485 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:27.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:27.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:28.179 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:28.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:29.042 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:29.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:29.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:29.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:30.564 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:30.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:30.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:31.218 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:31.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:32.047 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:32.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:32.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:32.802 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:32.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:33.529 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:33.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:33.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:34.291 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:34.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:35.060 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:35.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:35.081 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:35.822 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:35.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:36.567 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:36.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:36.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:37.369 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:37.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:38.122 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:38.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:38.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:38.926 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:38.950 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:39.670 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:39.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:17:39.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:40.423 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:40.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:41.192 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:41.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-04-25T09:17:41.264 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T09:17:41.264 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-04-25T09:17:41.264 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_460268 2024-04-25T09:17:41.265 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:17:41.265 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:17:42.264 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 193 177 0.172833 0.172852 0.0669201 0.0852693 2024-04-25T09:17:43.265 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 1.04073 2024-04-25T09:17:43.265 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 193 2024-04-25T09:17:43.265 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 1024 2024-04-25T09:17:43.265 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 1024 2024-04-25T09:17:43.265 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.181101 2024-04-25T09:17:43.265 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.172852 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.172852 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 185 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 0 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 177 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 177 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.0837362 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.022364 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.138821 2024-04-25T09:17:43.266 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.0660631 2024-04-25T09:17:43.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:17:43.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:17:43.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:17:43.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:17:43.302 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:17:43.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:17:43.303 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:17:43.303 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:17:43.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:17:43.304 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:17:43.430 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:17:43.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:17:43.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:17:43.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:17:43.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:17:43.431 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:17:43.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:17:43.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:17:43.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:43.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:17:44.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215126 2024-04-25T09:17:44.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215126 2024-04-25T09:17:44.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215126' 2024-04-25T09:17:44.087 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:44.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215126 2024-04-25T09:17:44.087 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:44.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:17:44.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215126 2024-04-25T09:17:44.089 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:44.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215126 2024-04-25T09:17:44.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215126' 2024-04-25T09:17:44.091 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 103079215126 2024-04-25T09:17:44.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:44.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215123 -lt 103079215126 2024-04-25T09:17:44.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:17:45.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:17:45.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:46.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215126 -lt 103079215126 2024-04-25T09:17:46.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:17:46.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:46.002 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:46.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:17:46.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:46.564 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:46.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:46.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:46.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:46.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:46.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:46.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:17:46.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:46.969 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:46.970 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:47.534 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:17:47.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:17:47.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:17:47.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-04-25T09:17:47.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:17:52.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:17:53.569 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is set 2024-04-25T09:17:53.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:17:54.313 INFO:tasks.workunit.client.0.smithi031.stderr:nodown is unset 2024-04-25T09:17:54.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:17:59.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:17:59.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:17:59.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:17:59.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:17:59.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:17:59.335 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:17:59.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:17:59.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:18:00.716 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:18:00.716 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.1 2024-04-25T09:18:00.716 INFO:tasks.workunit.client.0.smithi031.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:18:00.717 INFO:tasks.workunit.client.0.smithi031.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:18:01.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-04-25T09:18:02.210 INFO:tasks.workunit.client.0.smithi031.stderr:Exporting 1.0 info 1.0( v 225'407 (10'213,225'407] local-lis/les=24/25 n=595 ec=7/7 lis/c=24/24 les/c/f=25/25/0 sis=24) 2024-04-25T09:18:02.227 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0005830b:::benchmark_data_smithi031_454752_object169:head# 2024-04-25T09:18:02.228 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0007ff59:::benchmark_data_smithi031_454752_object384:head# 2024-04-25T09:18:02.228 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00530cad:::benchmark_data_smithi031_454752_object366:head# 2024-04-25T09:18:02.229 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:007182ad:::benchmark_data_smithi031_454752_object150:head# 2024-04-25T09:18:02.229 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00808219:::benchmark_data_smithi031_454752_object10:head# 2024-04-25T09:18:02.230 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00ccfbba:::benchmark_data_smithi031_454752_object355:head# 2024-04-25T09:18:02.230 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00d7eda8:::benchmark_data_smithi031_454752_object357:head# 2024-04-25T09:18:02.231 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:019d4e67:::benchmark_data_smithi031_460268_object76:head# 2024-04-25T09:18:02.231 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:01faf234:::benchmark_data_smithi031_454752_object272:head# 2024-04-25T09:18:02.232 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:021d5fe5:::benchmark_data_smithi031_454752_object109:head# 2024-04-25T09:18:02.232 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:023d725e:::benchmark_data_smithi031_460268_object88:head# 2024-04-25T09:18:02.233 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:02a87b95:::benchmark_data_smithi031_454752_object26:head# 2024-04-25T09:18:02.233 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:02bf8c5e:::benchmark_data_smithi031_460268_object64:head# 2024-04-25T09:18:02.234 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:03119e67:::benchmark_data_smithi031_454752_object120:head# 2024-04-25T09:18:02.234 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:032c7ba3:::benchmark_data_smithi031_460268_object73:head# 2024-04-25T09:18:02.235 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:03a19a01:::benchmark_data_smithi031_454752_object375:head# 2024-04-25T09:18:02.235 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:03d8d0d6:::benchmark_data_smithi031_454752_object307:head# 2024-04-25T09:18:02.236 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0407316c:::benchmark_data_smithi031_454752_object248:head# 2024-04-25T09:18:02.237 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:04e37833:::benchmark_data_smithi031_460268_object95:head# 2024-04-25T09:18:02.237 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:05c088cd:::benchmark_data_smithi031_460268_object132:head# 2024-04-25T09:18:02.237 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:069ac774:::benchmark_data_smithi031_454752_object270:head# 2024-04-25T09:18:02.238 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:076ac3b3:::benchmark_data_smithi031_454752_object168:head# 2024-04-25T09:18:02.238 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:078ca8f4:::benchmark_data_smithi031_454752_object271:head# 2024-04-25T09:18:02.242 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:089737ee:::benchmark_data_smithi031_454752_object262:head# 2024-04-25T09:18:02.243 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:08fea58a:::benchmark_data_smithi031_460268_object40:head# 2024-04-25T09:18:02.243 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0954f670:::benchmark_data_smithi031_454752_object223:head# 2024-04-25T09:18:02.244 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:09d319a2:::benchmark_data_smithi031_454752_object283:head# 2024-04-25T09:18:02.244 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0a33ee0a:::benchmark_data_smithi031_454752_object107:head# 2024-04-25T09:18:02.245 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0a588e5f:::benchmark_data_smithi031_454752_object305:head# 2024-04-25T09:18:02.245 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ae0c865:::benchmark_data_smithi031_454752_object41:head# 2024-04-25T09:18:02.247 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0c38e0a3:::benchmark_data_smithi031_454752_object387:head# 2024-04-25T09:18:02.247 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0c4decad:::benchmark_data_smithi031_454752_object151:head# 2024-04-25T09:18:02.247 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0cc7bcb7:::benchmark_data_smithi031_454752_object333:head# 2024-04-25T09:18:02.248 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ce7b7bc:::benchmark_data_smithi031_454752_object353:head# 2024-04-25T09:18:02.248 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0d6e3436:::benchmark_data_smithi031_454752_object205:head# 2024-04-25T09:18:02.249 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0dd158a0:::benchmark_data_smithi031_454752_object76:head# 2024-04-25T09:18:02.249 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0df905cb:::benchmark_data_smithi031_454752_object181:head# 2024-04-25T09:18:02.250 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0e56a830:::benchmark_data_smithi031_460268_object58:head# 2024-04-25T09:18:02.250 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ec7061f:::benchmark_data_smithi031_460268_object9:head# 2024-04-25T09:18:02.251 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0f047668:::benchmark_data_smithi031_454752_object12:head# 2024-04-25T09:18:02.252 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0f62c75d:::benchmark_data_smithi031_460268_object111:head# 2024-04-25T09:18:02.252 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:100b6e67:::benchmark_data_smithi031_454752_object122:head# 2024-04-25T09:18:02.253 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:105e9b29:::benchmark_data_smithi031_454752_object295:head# 2024-04-25T09:18:02.254 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:10ca342c:::benchmark_data_smithi031_454752_object203:head# 2024-04-25T09:18:02.254 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:10d8c220:::benchmark_data_smithi031_460268_object108:head# 2024-04-25T09:18:02.254 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:117584c9:::benchmark_data_smithi031_460268_object80:head# 2024-04-25T09:18:02.255 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:11985b73:::benchmark_data_smithi031_454752_object294:head# 2024-04-25T09:18:02.255 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1215ebb0:::benchmark_data_smithi031_454752_object96:head# 2024-04-25T09:18:02.256 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:123f25d6:::benchmark_data_smithi031_454752_object339:head# 2024-04-25T09:18:02.256 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:12648b62:::benchmark_data_smithi031_454752_object235:head# 2024-04-25T09:18:02.257 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:13b45992:::benchmark_data_smithi031_460268_object5:head# 2024-04-25T09:18:02.258 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:13b5933b:::benchmark_data_smithi031_454752_object258:head# 2024-04-25T09:18:02.258 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:14a4f03a:::benchmark_data_smithi031_454752_object90:head# 2024-04-25T09:18:02.258 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:14f470c1:::benchmark_data_smithi031_460268_object61:head# 2024-04-25T09:18:02.258 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:152756a2:::benchmark_data_smithi031_454752_object78:head# 2024-04-25T09:18:02.258 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:16b02634:::benchmark_data_smithi031_460268_object169:head# 2024-04-25T09:18:02.259 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:174b1785:::benchmark_data_smithi031_460268_object49:head# 2024-04-25T09:18:02.259 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:17685eef:::benchmark_data_smithi031_454752_object35:head# 2024-04-25T09:18:02.260 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:17d78309:::benchmark_data_smithi031_454752_object197:head# 2024-04-25T09:18:02.260 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:186323b7:::benchmark_data_smithi031_454752_object264:head# 2024-04-25T09:18:02.261 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18a96ee0:::benchmark_data_smithi031_454752_object340:head# 2024-04-25T09:18:02.261 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18abc847:::benchmark_data_smithi031_454752_object377:head# 2024-04-25T09:18:02.262 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:190ad911:::benchmark_data_smithi031_454752_object158:head# 2024-04-25T09:18:02.262 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:192b4b6c:::benchmark_data_smithi031_454752_object249:head# 2024-04-25T09:18:02.262 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:19e0e3b2:::benchmark_data_smithi031_460268_object175:head# 2024-04-25T09:18:02.263 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1a2dbf6a:::benchmark_data_smithi031_454752_object284:head# 2024-04-25T09:18:02.264 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1a523430:::benchmark_data_smithi031_460268_object184:head# 2024-04-25T09:18:02.264 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1ba28391:::benchmark_data_smithi031_460268_object129:head# 2024-04-25T09:18:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1bace3cc:::benchmark_data_smithi031_454752_object136:head# 2024-04-25T09:18:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1c390819:::benchmark_data_smithi031_460268_object149:head# 2024-04-25T09:18:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1da2bdd0:::benchmark_data_smithi031_460268_object168:head# 2024-04-25T09:18:02.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1e87b9f0:::benchmark_data_smithi031_460268_object130:head# 2024-04-25T09:18:02.266 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1f6172c3:::benchmark_data_smithi031_454752_object315:head# 2024-04-25T09:18:02.267 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1f9ba3a6:::benchmark_data_smithi031_460268_object191:head# 2024-04-25T09:18:02.268 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:20988891:::benchmark_data_smithi031_454752_object0:head# 2024-04-25T09:18:02.268 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:20c89311:::benchmark_data_smithi031_454752_object170:head# 2024-04-25T09:18:02.268 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:20f96ee6:::benchmark_data_smithi031_454752_object30:head# 2024-04-25T09:18:02.269 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:210b81be:::benchmark_data_smithi031_454752_object396:head# 2024-04-25T09:18:02.269 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:222bd3b2:::benchmark_data_smithi031_454752_object244:head# 2024-04-25T09:18:02.269 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:23439e48:::benchmark_data_smithi031_460268_object90:head# 2024-04-25T09:18:02.270 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:23ea9d14:::benchmark_data_smithi031_454752_object134:head# 2024-04-25T09:18:02.270 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2564bee6:::benchmark_data_smithi031_454752_object68:head# 2024-04-25T09:18:02.271 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2587b006:::benchmark_data_smithi031_460268_object114:head# 2024-04-25T09:18:02.272 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:259c03b1:::benchmark_data_smithi031_460268_object44:head# 2024-04-25T09:18:02.272 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:26e5d41e:::benchmark_data_smithi031_454752_object261:head# 2024-04-25T09:18:02.272 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2765dc2c:::benchmark_data_smithi031_454752_object147:head# 2024-04-25T09:18:02.273 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2805b1eb:::benchmark_data_smithi031_460268_object10:head# 2024-04-25T09:18:02.273 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2826ee59:::benchmark_data_smithi031_460268_object1:head# 2024-04-25T09:18:02.273 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:286a6080:::benchmark_data_smithi031_454752_object64:head# 2024-04-25T09:18:02.274 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2899a918:::benchmark_data_smithi031_454752_object267:head# 2024-04-25T09:18:02.274 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:293ed6b9:::benchmark_data_smithi031_460268_object55:head# 2024-04-25T09:18:02.275 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2972c7c4:::benchmark_data_smithi031_460268_object141:head# 2024-04-25T09:18:02.275 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:299c9ca7:::benchmark_data_smithi031_454752_object21:head# 2024-04-25T09:18:02.275 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:29db69da:::benchmark_data_smithi031_454752_object236:head# 2024-04-25T09:18:02.276 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:29dfd43b:::benchmark_data_smithi031_460268_object144:head# 2024-04-25T09:18:02.276 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2aa178ac:::benchmark_data_smithi031_454752_object59:head# 2024-04-25T09:18:02.277 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2ac07948:::benchmark_data_smithi031_454752_object7:head# 2024-04-25T09:18:02.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2af7345c:::benchmark_data_smithi031_454752_object202:head# 2024-04-25T09:18:02.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2bd9629a:::benchmark_data_smithi031_460268_object68:head# 2024-04-25T09:18:02.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2c7cae34:::benchmark_data_smithi031_454752_object300:head# 2024-04-25T09:18:02.279 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2d714369:::benchmark_data_smithi031_454752_object292:head# 2024-04-25T09:18:02.279 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2da27532:::benchmark_data_smithi031_454752_object1:head# 2024-04-25T09:18:02.280 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2dde69f0:::benchmark_data_smithi031_460268_object103:head# 2024-04-25T09:18:02.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2dee9738:::benchmark_data_smithi031_454752_object400:head# 2024-04-25T09:18:02.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2e578dfe:::benchmark_data_smithi031_460268_object86:head# 2024-04-25T09:18:02.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2e61c031:::benchmark_data_smithi031_454752_object174:head# 2024-04-25T09:18:02.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2ee4a5f6:::benchmark_data_smithi031_454752_object94:head# 2024-04-25T09:18:02.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2f1e460a:::benchmark_data_smithi031_454752_object106:head# 2024-04-25T09:18:02.282 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2f48312c:::benchmark_data_smithi031_460268_object127:head# 2024-04-25T09:18:02.283 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2fb10bd8:::benchmark_data_smithi031_460268_object159:head# 2024-04-25T09:18:02.283 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2fc8a11e:::benchmark_data_smithi031_454752_object260:head# 2024-04-25T09:18:02.283 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2fcd61f1:::benchmark_data_smithi031_460268_object140:head# 2024-04-25T09:18:02.284 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:30797560:::benchmark_data_smithi031_454752_object142:head# 2024-04-25T09:18:02.284 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3105618d:::benchmark_data_smithi031_454752_object46:head# 2024-04-25T09:18:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3136920f:::benchmark_data_smithi031_460268_object131:head# 2024-04-25T09:18:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31a53b00:::benchmark_data_smithi031_454752_object14:head# 2024-04-25T09:18:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31b8e0ab:::benchmark_data_smithi031_454752_object326:head# 2024-04-25T09:18:02.286 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31c961a5:::benchmark_data_smithi031_454752_object11:head# 2024-04-25T09:18:02.286 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31f4feb6:::benchmark_data_smithi031_454752_object54:head# 2024-04-25T09:18:02.287 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:32147e98:::benchmark_data_smithi031_460268_object151:head# 2024-04-25T09:18:02.287 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3295aa64:::benchmark_data_smithi031_460268_object46:head# 2024-04-25T09:18:02.288 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:332a7b27:::benchmark_data_smithi031_454752_object257:head# 2024-04-25T09:18:02.288 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:33cf1783:::benchmark_data_smithi031_454752_object83:head# 2024-04-25T09:18:02.288 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:33e69519:::benchmark_data_smithi031_454752_object334:head# 2024-04-25T09:18:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:343b464c:::benchmark_data_smithi031_454752_object229:head# 2024-04-25T09:18:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:34fc845a:::benchmark_data_smithi031_460268_object146:head# 2024-04-25T09:18:02.289 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:353bc8d6:::benchmark_data_smithi031_454752_object306:head# 2024-04-25T09:18:02.290 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:355e1d34:::benchmark_data_smithi031_460268_object75:head# 2024-04-25T09:18:02.290 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:357adf2e:::benchmark_data_smithi031_454752_object313:head# 2024-04-25T09:18:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3626a3d2:::benchmark_data_smithi031_454752_object242:head# 2024-04-25T09:18:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:366a3868:::benchmark_data_smithi031_454752_object344:head# 2024-04-25T09:18:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:36e1e4b7:::benchmark_data_smithi031_454752_object330:head# 2024-04-25T09:18:02.292 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3747f9e2:::benchmark_data_smithi031_454752_object361:head# 2024-04-25T09:18:02.293 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:376c5fd1:::benchmark_data_smithi031_454752_object276:head# 2024-04-25T09:18:02.293 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:37ad6f44:::benchmark_data_smithi031_454752_object199:head# 2024-04-25T09:18:02.293 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3809a39d:::benchmark_data_smithi031_454752_object63:head# 2024-04-25T09:18:02.293 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3861888a:::benchmark_data_smithi031_460268_object109:head# 2024-04-25T09:18:02.294 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:38962184:::benchmark_data_smithi031_454752_object345:head# 2024-04-25T09:18:02.295 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3947aa59:::benchmark_data_smithi031_454752_object29:head# 2024-04-25T09:18:02.295 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:39a62bf7:::benchmark_data_smithi031_454752_object346:head# 2024-04-25T09:18:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3a70fad2:::benchmark_data_smithi031_454752_object213:head# 2024-04-25T09:18:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3aaccaa2:::benchmark_data_smithi031_454752_object245:head# 2024-04-25T09:18:02.296 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3acaee8b:::benchmark_data_smithi031_460268_object48:head# 2024-04-25T09:18:02.297 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3b4d69f0:::benchmark_data_smithi031_454752_object133:head# 2024-04-25T09:18:02.298 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3b5ed4ad:::benchmark_data_smithi031_454752_object153:head# 2024-04-25T09:18:02.298 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3bd7c293:::benchmark_data_smithi031_454752_object226:head# 2024-04-25T09:18:02.299 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3d9bbce6:::benchmark_data_smithi031_454752_object195:head# 2024-04-25T09:18:02.299 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3e1dceb1:::benchmark_data_smithi031_454752_object163:head# 2024-04-25T09:18:02.300 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3f585de7:::benchmark_data_smithi031_454752_object370:head# 2024-04-25T09:18:02.300 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3f8dddc0:::benchmark_data_smithi031_454752_object36:head# 2024-04-25T09:18:02.300 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:40e869df:::benchmark_data_smithi031_454752_object354:head# 2024-04-25T09:18:02.300 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:42077408:::benchmark_data_smithi031_460268_object138:head# 2024-04-25T09:18:02.301 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4240854c:::benchmark_data_smithi031_460268_object99:head# 2024-04-25T09:18:02.302 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:429d8266:::benchmark_data_smithi031_460268_object157:head# 2024-04-25T09:18:02.302 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:42f70b52:::benchmark_data_smithi031_454752_object391:head# 2024-04-25T09:18:02.303 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:42fae8c1:::benchmark_data_smithi031_460268_object158:head# 2024-04-25T09:18:02.304 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:44648b7f:::benchmark_data_smithi031_454752_object15:head# 2024-04-25T09:18:02.304 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:44e34ed7:::benchmark_data_smithi031_460268_object47:head# 2024-04-25T09:18:02.304 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:45c0069e:::benchmark_data_smithi031_454752_object93:head# 2024-04-25T09:18:02.304 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:45d9ab4f:::benchmark_data_smithi031_460268_object79:head# 2024-04-25T09:18:02.305 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:46cce1fb:::benchmark_data_smithi031_454752_object259:head# 2024-04-25T09:18:02.306 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:48372743:::benchmark_data_smithi031_454752_object92:head# 2024-04-25T09:18:02.306 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:487de23f:::benchmark_data_smithi031_454752_object278:head# 2024-04-25T09:18:02.306 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:48933b4f:::benchmark_data_smithi031_454752_object219:head# 2024-04-25T09:18:02.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:48b43814:::benchmark_data_smithi031_454752_object22:head# 2024-04-25T09:18:02.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:49487ec1:::benchmark_data_smithi031_460268_object62:head# 2024-04-25T09:18:02.308 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:49675aeb:::benchmark_data_smithi031_454752_object99:head# 2024-04-25T09:18:02.308 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a2d5412:::benchmark_data_smithi031_454752_object324:head# 2024-04-25T09:18:02.309 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a6e0d41:::benchmark_data_smithi031_454752_object268:head# 2024-04-25T09:18:02.309 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a6e92f8:::benchmark_data_smithi031_454752_object198:head# 2024-04-25T09:18:02.310 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a95c670:::benchmark_data_smithi031_454752_object220:head# 2024-04-25T09:18:02.310 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4a9a32d5:::benchmark_data_smithi031_454752_object56:head# 2024-04-25T09:18:02.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4aa2f1b1:::benchmark_data_smithi031_454752_object160:head# 2024-04-25T09:18:02.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4ac6a300:::benchmark_data_smithi031_454752_object209:head# 2024-04-25T09:18:02.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4af7d290:::benchmark_data_smithi031_460268_object97:head# 2024-04-25T09:18:02.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b065b88:::benchmark_data_smithi031_454752_object34:head# 2024-04-25T09:18:02.313 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b199cf4:::benchmark_data_smithi031_460268_object102:head# 2024-04-25T09:18:02.313 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b3ae5fa:::benchmark_data_smithi031_454752_object399:head# 2024-04-25T09:18:02.313 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b491b4d:::benchmark_data_smithi031_454752_object86:head# 2024-04-25T09:18:02.314 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b70dd42:::benchmark_data_smithi031_460268_object133:head# 2024-04-25T09:18:02.314 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4c281e67:::benchmark_data_smithi031_454752_object123:head# 2024-04-25T09:18:02.315 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cc52962:::benchmark_data_smithi031_460268_object122:head# 2024-04-25T09:18:02.315 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cd1274c:::benchmark_data_smithi031_454752_object77:head# 2024-04-25T09:18:02.316 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cd91fcf:::benchmark_data_smithi031_454752_object227:head# 2024-04-25T09:18:02.316 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4cd9e311:::benchmark_data_smithi031_454752_object165:head# 2024-04-25T09:18:02.316 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4e704b66:::benchmark_data_smithi031_454752_object110:head# 2024-04-25T09:18:02.317 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4f141352:::benchmark_data_smithi031_454752_object390:head# 2024-04-25T09:18:02.318 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4f67372e:::benchmark_data_smithi031_454752_object310:head# 2024-04-25T09:18:02.318 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5055bf66:::benchmark_data_smithi031_454752_object111:head# 2024-04-25T09:18:02.318 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:505bbf46:::benchmark_data_smithi031_460268_object180:head# 2024-04-25T09:18:02.319 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5079fdc4:::benchmark_data_smithi031_454752_object102:head# 2024-04-25T09:18:02.320 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51592368:::benchmark_data_smithi031_454752_object129:head# 2024-04-25T09:18:02.320 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51c35831:::benchmark_data_smithi031_454752_object277:head# 2024-04-25T09:18:02.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:525ef634:::benchmark_data_smithi031_454752_object303:head# 2024-04-25T09:18:02.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:526ebac4:::benchmark_data_smithi031_460268_object172:head# 2024-04-25T09:18:02.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:52f1a683:::benchmark_data_smithi031_454752_object232:head# 2024-04-25T09:18:02.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:539a5e15:::benchmark_data_smithi031_460268_object96:head# 2024-04-25T09:18:02.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:53c14896:::benchmark_data_smithi031_460268_object29:head# 2024-04-25T09:18:02.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:549943cc:::benchmark_data_smithi031_454752_object137:head# 2024-04-25T09:18:02.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:54a7cad2:::benchmark_data_smithi031_454752_object243:head# 2024-04-25T09:18:02.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56294a42:::benchmark_data_smithi031_460268_object125:head# 2024-04-25T09:18:02.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:565ee0b3:::benchmark_data_smithi031_460268_object87:head# 2024-04-25T09:18:02.325 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56c9c8a3:::benchmark_data_smithi031_454752_object386:head# 2024-04-25T09:18:02.325 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56d22a80:::benchmark_data_smithi031_454752_object61:head# 2024-04-25T09:18:02.326 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57498d87:::benchmark_data_smithi031_460268_object15:head# 2024-04-25T09:18:02.326 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57503a76:::benchmark_data_smithi031_460268_object177:head# 2024-04-25T09:18:02.326 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:576c3dbd:::benchmark_data_smithi031_454752_object320:head# 2024-04-25T09:18:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57e97d52:::benchmark_data_smithi031_460268_object152:head# 2024-04-25T09:18:02.328 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57f53672:::benchmark_data_smithi031_460268_object137:head# 2024-04-25T09:18:02.329 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:580c5a46:::benchmark_data_smithi031_454752_object62:head# 2024-04-25T09:18:02.329 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:58815a7c:::benchmark_data_smithi031_454752_object392:head# 2024-04-25T09:18:02.329 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:58e6f983:::benchmark_data_smithi031_454752_object233:head# 2024-04-25T09:18:02.330 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:59c109e7:::benchmark_data_smithi031_454752_object372:head# 2024-04-25T09:18:02.331 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5a0f0b43:::benchmark_data_smithi031_460268_object19:head# 2024-04-25T09:18:02.331 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5a7c4e16:::benchmark_data_smithi031_454752_object48:head# 2024-04-25T09:18:02.331 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5a943612:::benchmark_data_smithi031_454752_object40:head# 2024-04-25T09:18:02.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5aed2644:::benchmark_data_smithi031_454752_object395:head# 2024-04-25T09:18:02.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5b4543d1:::benchmark_data_smithi031_460268_object98:head# 2024-04-25T09:18:02.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5bae9bea:::benchmark_data_smithi031_460268_object165:head# 2024-04-25T09:18:02.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5c81318e:::benchmark_data_smithi031_454752_object18:head# 2024-04-25T09:18:02.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5c9e0861:::benchmark_data_smithi031_460268_object178:head# 2024-04-25T09:18:02.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5cde96ce:::benchmark_data_smithi031_460268_object164:head# 2024-04-25T09:18:02.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5d21802c:::benchmark_data_smithi031_454752_object201:head# 2024-04-25T09:18:02.335 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5d50b842:::benchmark_data_smithi031_454752_object282:head# 2024-04-25T09:18:02.335 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5de1e511:::benchmark_data_smithi031_454752_object173:head# 2024-04-25T09:18:02.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e1e382c:::benchmark_data_smithi031_454752_object189:head# 2024-04-25T09:18:02.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e3603a8:::benchmark_data_smithi031_454752_object356:head# 2024-04-25T09:18:02.337 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e9091db:::benchmark_data_smithi031_460268_object41:head# 2024-04-25T09:18:02.337 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e9a3526:::benchmark_data_smithi031_454752_object274:head# 2024-04-25T09:18:02.338 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5f2d69c6:::benchmark_data_smithi031_460268_object0:head# 2024-04-25T09:18:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5fb49c8b:::benchmark_data_smithi031_460268_object160:head# 2024-04-25T09:18:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:60047f02:::benchmark_data_smithi031_460268_object69:head# 2024-04-25T09:18:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:60dee496:::benchmark_data_smithi031_454752_object287:head# 2024-04-25T09:18:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:610072df:::benchmark_data_smithi031_454752_object381:head# 2024-04-25T09:18:02.340 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:61757bc4:::benchmark_data_smithi031_454752_object100:head# 2024-04-25T09:18:02.341 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:61eb06a1:::benchmark_data_smithi031_454752_object42:head# 2024-04-25T09:18:02.341 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:621c25df:::benchmark_data_smithi031_460268_object123:head# 2024-04-25T09:18:02.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:622a095a:::benchmark_data_smithi031_454752_object172:head# 2024-04-25T09:18:02.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:629696e0:::benchmark_data_smithi031_454752_object343:head# 2024-04-25T09:18:02.343 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:62b15a24:::benchmark_data_smithi031_460268_object107:head# 2024-04-25T09:18:02.343 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:62dff125:::benchmark_data_smithi031_454752_object117:head# 2024-04-25T09:18:02.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:637e2a8b:::benchmark_data_smithi031_454752_object124:head# 2024-04-25T09:18:02.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:63a84c81:::benchmark_data_smithi031_454752_object82:head# 2024-04-25T09:18:02.345 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:645c6c7a:::benchmark_data_smithi031_460268_object60:head# 2024-04-25T09:18:02.345 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:64a1f0ad:::benchmark_data_smithi031_454752_object367:head# 2024-04-25T09:18:02.346 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:650a3266:::benchmark_data_smithi031_460268_object77:head# 2024-04-25T09:18:02.346 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6554a2d5:::benchmark_data_smithi031_460268_object37:head# 2024-04-25T09:18:02.347 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:65a082ea:::benchmark_data_smithi031_454752_object316:head# 2024-04-25T09:18:02.347 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:66d0a0c5:::benchmark_data_smithi031_454752_object4:head# 2024-04-25T09:18:02.347 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:671a3d79:::benchmark_data_smithi031_460268_object70:head# 2024-04-25T09:18:02.349 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:675475fc:::benchmark_data_smithi031_460268_object143:head# 2024-04-25T09:18:02.350 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6757b3f7:::benchmark_data_smithi031_454752_object347:head# 2024-04-25T09:18:02.350 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:67d2de21:::benchmark_data_smithi031_460268_object11:head# 2024-04-25T09:18:02.350 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:68851dbf:::benchmark_data_smithi031_454752_object279:head# 2024-04-25T09:18:02.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:68869d9f:::benchmark_data_smithi031_454752_object8:head# 2024-04-25T09:18:02.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:68a2186c:::benchmark_data_smithi031_460268_object121:head# 2024-04-25T09:18:02.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6922a669:::benchmark_data_smithi031_454752_object291:head# 2024-04-25T09:18:02.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:695ddfd2:::benchmark_data_smithi031_454752_object285:head# 2024-04-25T09:18:02.352 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:69f748df:::benchmark_data_smithi031_454752_object380:head# 2024-04-25T09:18:02.352 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:69fb4598:::benchmark_data_smithi031_460268_object176:head# 2024-04-25T09:18:02.352 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6a4d4ed2:::benchmark_data_smithi031_454752_object212:head# 2024-04-25T09:18:02.352 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6b91f5bc:::benchmark_data_smithi031_460268_object155:head# 2024-04-25T09:18:02.353 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ca35829:::benchmark_data_smithi031_460268_object167:head# 2024-04-25T09:18:02.353 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ce64eae:::benchmark_data_smithi031_460268_object21:head# 2024-04-25T09:18:02.354 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6d59ce53:::benchmark_data_smithi031_460268_object54:head# 2024-04-25T09:18:02.354 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6d7df3bd:::benchmark_data_smithi031_454752_object323:head# 2024-04-25T09:18:02.355 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6d9bcab7:::benchmark_data_smithi031_454752_object299:head# 2024-04-25T09:18:02.355 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6e1dd161:::benchmark_data_smithi031_454752_object176:head# 2024-04-25T09:18:02.356 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6e68daeb:::benchmark_data_smithi031_460268_object135:head# 2024-04-25T09:18:02.356 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ea26cda:::benchmark_data_smithi031_454752_object237:head# 2024-04-25T09:18:02.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6f7ed7e9:::benchmark_data_smithi031_460268_object26:head# 2024-04-25T09:18:02.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6f89a95c:::benchmark_data_smithi031_460268_object148:head# 2024-04-25T09:18:02.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:70e2520b:::benchmark_data_smithi031_460268_object170:head# 2024-04-25T09:18:02.358 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7146e359:::benchmark_data_smithi031_460268_object116:head# 2024-04-25T09:18:02.358 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:715f84d9:::benchmark_data_smithi031_454752_object31:head# 2024-04-25T09:18:02.359 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:718b1888:::benchmark_data_smithi031_454752_object3:head# 2024-04-25T09:18:02.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7201ba57:::benchmark_data_smithi031_454752_object332:head# 2024-04-25T09:18:02.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:726d7c01:::benchmark_data_smithi031_460268_object89:head# 2024-04-25T09:18:02.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72916a4f:::benchmark_data_smithi031_454752_object43:head# 2024-04-25T09:18:02.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72e5e7bc:::benchmark_data_smithi031_454752_object350:head# 2024-04-25T09:18:02.362 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72f2ccd2:::benchmark_data_smithi031_454752_object211:head# 2024-04-25T09:18:02.362 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:734a5a75:::benchmark_data_smithi031_460268_object2:head# 2024-04-25T09:18:02.363 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:737dcd92:::benchmark_data_smithi031_454752_object335:head# 2024-04-25T09:18:02.363 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:744eb9c4:::benchmark_data_smithi031_454752_object101:head# 2024-04-25T09:18:02.363 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:749ce32a:::benchmark_data_smithi031_460268_object142:head# 2024-04-25T09:18:02.364 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:74d52557:::benchmark_data_smithi031_460268_object56:head# 2024-04-25T09:18:02.365 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:750b59a6:::benchmark_data_smithi031_460268_object85:head# 2024-04-25T09:18:02.365 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:75451e3a:::benchmark_data_smithi031_454752_object215:head# 2024-04-25T09:18:02.365 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:75b87d24:::benchmark_data_smithi031_454752_object156:head# 2024-04-25T09:18:02.366 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:76a44be7:::benchmark_data_smithi031_454752_object371:head# 2024-04-25T09:18:02.366 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:76a5bace:::benchmark_data_smithi031_454752_object253:head# 2024-04-25T09:18:02.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:772eee67:::benchmark_data_smithi031_454752_object121:head# 2024-04-25T09:18:02.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:78512cf7:::benchmark_data_smithi031_460268_object139:head# 2024-04-25T09:18:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:78a2ff3a:::benchmark_data_smithi031_460268_object162:head# 2024-04-25T09:18:02.368 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79141f23:::benchmark_data_smithi031_460268_object78:head# 2024-04-25T09:18:02.369 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:798c5654:::benchmark_data_smithi031_454752_object204:head# 2024-04-25T09:18:02.369 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7995db24:::benchmark_data_smithi031_454752_object157:head# 2024-04-25T09:18:02.371 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79abc611:::benchmark_data_smithi031_454752_object164:head# 2024-04-25T09:18:02.371 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79d1490c:::benchmark_data_smithi031_454752_object138:head# 2024-04-25T09:18:02.372 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7adc14de:::benchmark_data_smithi031_460268_object43:head# 2024-04-25T09:18:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7b6edc17:::benchmark_data_smithi031_454752_object214:head# 2024-04-25T09:18:02.373 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7b93ca23:::benchmark_data_smithi031_454752_object55:head# 2024-04-25T09:18:02.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7ba8e7d6:::benchmark_data_smithi031_460268_object67:head# 2024-04-25T09:18:02.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bd5320a:::benchmark_data_smithi031_454752_object328:head# 2024-04-25T09:18:02.375 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bde86cb:::benchmark_data_smithi031_454752_object182:head# 2024-04-25T09:18:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bfed9be:::benchmark_data_smithi031_454752_object397:head# 2024-04-25T09:18:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7c002612:::benchmark_data_smithi031_460268_object110:head# 2024-04-25T09:18:02.376 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7c0b3015:::benchmark_data_smithi031_454752_object108:head# 2024-04-25T09:18:02.377 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7cadc342:::benchmark_data_smithi031_460268_object18:head# 2024-04-25T09:18:02.377 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7cb420f8:::benchmark_data_smithi031_454752_object89:head# 2024-04-25T09:18:02.378 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7d588f87:::benchmark_data_smithi031_454752_object376:head# 2024-04-25T09:18:02.378 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dbcb22e:::benchmark_data_smithi031_454752_object17:head# 2024-04-25T09:18:02.378 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dc6b3de:::benchmark_data_smithi031_454752_object187:head# 2024-04-25T09:18:02.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dec0b89:::benchmark_data_smithi031_454752_object149:head# 2024-04-25T09:18:02.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dfe5ddd:::benchmark_data_smithi031_454752_object79:head# 2024-04-25T09:18:02.380 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7ea5a1e2:::benchmark_data_smithi031_454752_object360:head# 2024-04-25T09:18:02.380 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7eeb5bde:::benchmark_data_smithi031_454752_object186:head# 2024-04-25T09:18:02.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7fb12f34:::benchmark_data_smithi031_454752_object69:head# 2024-04-25T09:18:02.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:800bce12:::benchmark_data_smithi031_460268_object118:head# 2024-04-25T09:18:02.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:80100b56:::benchmark_data_smithi031_460268_object22:head# 2024-04-25T09:18:02.382 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8187e1d1:::benchmark_data_smithi031_460268_object187:head# 2024-04-25T09:18:02.383 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:81a2bad0:::benchmark_data_smithi031_460268_object23:head# 2024-04-25T09:18:02.383 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:822ea472:::benchmark_data_smithi031_460268_object190:head# 2024-04-25T09:18:02.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:825fce9d:::benchmark_data_smithi031_454752_object9:head# 2024-04-25T09:18:02.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:82fe0341:::benchmark_data_smithi031_454752_object167:head# 2024-04-25T09:18:02.385 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:830b3926:::benchmark_data_smithi031_454752_object175:head# 2024-04-25T09:18:02.385 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:839ff2c1:::benchmark_data_smithi031_454752_object178:head# 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:83bed9ab:::benchmark_data_smithi031_454752_object192:head# 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:84bae7de:::benchmark_data_smithi031_454752_object19:head# 2024-04-25T09:18:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8514352c:::benchmark_data_smithi031_454752_object84:head# 2024-04-25T09:18:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:85dc3ddd:::benchmark_data_smithi031_460268_object92:head# 2024-04-25T09:18:02.387 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:85e422cb:::benchmark_data_smithi031_454752_object183:head# 2024-04-25T09:18:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8696227e:::benchmark_data_smithi031_454752_object304:head# 2024-04-25T09:18:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:869e69b7:::benchmark_data_smithi031_460268_object189:head# 2024-04-25T09:18:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:86aa36c1:::benchmark_data_smithi031_454752_object179:head# 2024-04-25T09:18:02.389 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:878ba262:::benchmark_data_smithi031_454752_object281:head# 2024-04-25T09:18:02.389 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:885c852e:::benchmark_data_smithi031_460268_object186:head# 2024-04-25T09:18:02.390 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:88d66484:::benchmark_data_smithi031_460268_object35:head# 2024-04-25T09:18:02.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:891fd6e5:::benchmark_data_smithi031_454752_object28:head# 2024-04-25T09:18:02.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:893b4bab:::benchmark_data_smithi031_460268_object113:head# 2024-04-25T09:18:02.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8941388a:::benchmark_data_smithi031_454752_object87:head# 2024-04-25T09:18:02.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:89e1ebba:::benchmark_data_smithi031_460268_object154:head# 2024-04-25T09:18:02.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8a4773c4:::benchmark_data_smithi031_454752_object103:head# 2024-04-25T09:18:02.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8aae6d43:::benchmark_data_smithi031_460268_object25:head# 2024-04-25T09:18:02.393 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8abbf253:::benchmark_data_smithi031_454752_object97:head# 2024-04-25T09:18:02.393 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8adf42cb:::benchmark_data_smithi031_454752_object180:head# 2024-04-25T09:18:02.393 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8ae888df:::benchmark_data_smithi031_454752_object382:head# 2024-04-25T09:18:02.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8b291374:::benchmark_data_smithi031_454752_object273:head# 2024-04-25T09:18:02.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8b621052:::benchmark_data_smithi031_454752_object368:head# 2024-04-25T09:18:02.395 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8c90b5b1:::benchmark_data_smithi031_454752_object161:head# 2024-04-25T09:18:02.395 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8d29fb8d:::benchmark_data_smithi031_454752_object327:head# 2024-04-25T09:18:02.395 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8d40e291:::benchmark_data_smithi031_460268_object120:head# 2024-04-25T09:18:02.396 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8db4e958:::benchmark_data_smithi031_454752_object385:head# 2024-04-25T09:18:02.396 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8eb4eb50:::benchmark_data_smithi031_460268_object8:head# 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8f54b87d:::benchmark_data_smithi031_460268_object13:head# 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8fa26b27:::benchmark_data_smithi031_454752_object256:head# 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:90588a1a:::benchmark_data_smithi031_454752_object378:head# 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:907e8918:::benchmark_data_smithi031_454752_object266:head# 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:924cae69:::benchmark_data_smithi031_460268_object12:head# 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9277f059:::benchmark_data_smithi031_454752_object2:head# 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:933ca87a:::benchmark_data_smithi031_460268_object181:head# 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:93d7997f:::benchmark_data_smithi031_460268_object6:head# 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:93ded35d:::benchmark_data_smithi031_460268_object59:head# 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9415aee4:::benchmark_data_smithi031_460268_object173:head# 2024-04-25T09:18:02.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:946e6356:::benchmark_data_smithi031_454752_object338:head# 2024-04-25T09:18:02.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:947d7692:::benchmark_data_smithi031_460268_object39:head# 2024-04-25T09:18:02.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9586170f:::benchmark_data_smithi031_454752_object13:head# 2024-04-25T09:18:02.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:95c08ad2:::benchmark_data_smithi031_454752_object210:head# 2024-04-25T09:18:02.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:95cd8cdf:::benchmark_data_smithi031_454752_object383:head# 2024-04-25T09:18:02.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9609476c:::benchmark_data_smithi031_454752_object114:head# 2024-04-25T09:18:02.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:967bc283:::benchmark_data_smithi031_454752_object231:head# 2024-04-25T09:18:02.404 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96801ae8:::benchmark_data_smithi031_454752_object207:head# 2024-04-25T09:18:02.404 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96a964d2:::benchmark_data_smithi031_454752_object38:head# 2024-04-25T09:18:02.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96aa787d:::benchmark_data_smithi031_454752_object322:head# 2024-04-25T09:18:02.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96d63edc:::benchmark_data_smithi031_454752_object352:head# 2024-04-25T09:18:02.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:97138c7c:::benchmark_data_smithi031_460268_object179:head# 2024-04-25T09:18:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:98e8b5bf:::benchmark_data_smithi031_454752_object216:head# 2024-04-25T09:18:02.406 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:99e3add0:::benchmark_data_smithi031_454752_object5:head# 2024-04-25T09:18:02.407 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9a07c74e:::benchmark_data_smithi031_454752_object252:head# 2024-04-25T09:18:02.407 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9a72fd5f:::benchmark_data_smithi031_454752_object297:head# 2024-04-25T09:18:02.408 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9a9f4dbd:::benchmark_data_smithi031_454752_object321:head# 2024-04-25T09:18:02.408 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9b86cf51:::benchmark_data_smithi031_454752_object225:head# 2024-04-25T09:18:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9be87125:::benchmark_data_smithi031_454752_object116:head# 2024-04-25T09:18:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9bf0e6c9:::benchmark_data_smithi031_454752_object47:head# 2024-04-25T09:18:02.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9c836d36:::benchmark_data_smithi031_454752_object44:head# 2024-04-25T09:18:02.410 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9cb1d2e2:::benchmark_data_smithi031_454752_object362:head# 2024-04-25T09:18:02.410 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9cbe41c2:::benchmark_data_smithi031_454752_object374:head# 2024-04-25T09:18:02.410 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9ce31ece:::benchmark_data_smithi031_454752_object308:head# 2024-04-25T09:18:02.411 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9df3f789:::benchmark_data_smithi031_454752_object394:head# 2024-04-25T09:18:02.411 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9e69968b:::benchmark_data_smithi031_460268_object126:head# 2024-04-25T09:18:02.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9e952085:::benchmark_data_smithi031_454752_object154:head# 2024-04-25T09:18:02.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9e95fdb1:::benchmark_data_smithi031_460268_object14:head# 2024-04-25T09:18:02.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9f2a7ece:::benchmark_data_smithi031_454752_object251:head# 2024-04-25T09:18:02.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a114ad36:::benchmark_data_smithi031_454752_object70:head# 2024-04-25T09:18:02.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a1535204:::benchmark_data_smithi031_460268_object71:head# 2024-04-25T09:18:02.414 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a2b210c5:::benchmark_data_smithi031_460268_object147:head# 2024-04-25T09:18:02.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a419031b:::benchmark_data_smithi031_454752_object81:head# 2024-04-25T09:18:02.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a45f5752:::benchmark_data_smithi031_454752_object39:head# 2024-04-25T09:18:02.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a46e3cab:::benchmark_data_smithi031_454752_object193:head# 2024-04-25T09:18:02.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a46e59f0:::benchmark_data_smithi031_454752_object130:head# 2024-04-25T09:18:02.416 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a47669cd:::benchmark_data_smithi031_460268_object82:head# 2024-04-25T09:18:02.417 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a4e9c0ae:::benchmark_data_smithi031_460268_object163:head# 2024-04-25T09:18:02.417 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a6bdfc76:::benchmark_data_smithi031_460268_object66:head# 2024-04-25T09:18:02.417 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a75374ab:::benchmark_data_smithi031_454752_object190:head# 2024-04-25T09:18:02.417 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a7706eea:::benchmark_data_smithi031_460268_object16:head# 2024-04-25T09:18:02.419 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a8271a67:::benchmark_data_smithi031_454752_object33:head# 2024-04-25T09:18:02.419 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a853f81a:::benchmark_data_smithi031_454752_object379:head# 2024-04-25T09:18:02.419 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a8c6bb46:::benchmark_data_smithi031_454752_object72:head# 2024-04-25T09:18:02.419 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a931c34a:::benchmark_data_smithi031_460268_object27:head# 2024-04-25T09:18:02.420 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a942e3ba:::benchmark_data_smithi031_454752_object74:head# 2024-04-25T09:18:02.421 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a9870539:::benchmark_data_smithi031_460268_object83:head# 2024-04-25T09:18:02.421 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a98c4d7c:::benchmark_data_smithi031_454752_object37:head# 2024-04-25T09:18:02.421 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a9e307d2:::benchmark_data_smithi031_454752_object318:head# 2024-04-25T09:18:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aab383a6:::benchmark_data_smithi031_454752_object112:head# 2024-04-25T09:18:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aaf58300:::benchmark_data_smithi031_454752_object208:head# 2024-04-25T09:18:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab0f7dc2:::benchmark_data_smithi031_454752_object119:head# 2024-04-25T09:18:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab1cf81e:::benchmark_data_smithi031_454752_object263:head# 2024-04-25T09:18:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab49749c:::benchmark_data_smithi031_454752_object359:head# 2024-04-25T09:18:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ac1d5d1f:::benchmark_data_smithi031_454752_object135:head# 2024-04-25T09:18:02.424 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ac7ea5bf:::benchmark_data_smithi031_454752_object217:head# 2024-04-25T09:18:02.424 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:acc3e569:::benchmark_data_smithi031_454752_object290:head# 2024-04-25T09:18:02.430 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ad398d0f:::benchmark_data_smithi031_454752_object125:head# 2024-04-25T09:18:02.430 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ae91e1ef:::benchmark_data_smithi031_454752_object115:head# 2024-04-25T09:18:02.431 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aea3dfa5:::benchmark_data_smithi031_454752_object388:head# 2024-04-25T09:18:02.431 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aec2dd45:::benchmark_data_smithi031_454752_object275:head# 2024-04-25T09:18:02.431 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aecd0faf:::benchmark_data_smithi031_460268_object156:head# 2024-04-25T09:18:02.433 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:af668e34:::benchmark_data_smithi031_454752_object302:head# 2024-04-25T09:18:02.433 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:af9a9573:::benchmark_data_smithi031_454752_object58:head# 2024-04-25T09:18:02.434 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:afcab8ff:::benchmark_data_smithi031_460268_object106:head# 2024-04-25T09:18:02.435 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b089bb4f:::benchmark_data_smithi031_454752_object218:head# 2024-04-25T09:18:02.435 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b1674de8:::benchmark_data_smithi031_454752_object194:head# 2024-04-25T09:18:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b2930ee9:::benchmark_data_smithi031_454752_object73:head# 2024-04-25T09:18:02.436 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b329d82c:::benchmark_data_smithi031_454752_object200:head# 2024-04-25T09:18:02.437 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b365eb28:::benchmark_data_smithi031_460268_object32:head# 2024-04-25T09:18:02.437 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b3c75192:::benchmark_data_smithi031_454752_object325:head# 2024-04-25T09:18:02.438 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b45e9911:::benchmark_data_smithi031_454752_object53:head# 2024-04-25T09:18:02.438 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b4f7e75f:::benchmark_data_smithi031_454752_object296:head# 2024-04-25T09:18:02.438 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b51eab52:::benchmark_data_smithi031_454752_object393:head# 2024-04-25T09:18:02.439 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b54079d2:::benchmark_data_smithi031_454752_object319:head# 2024-04-25T09:18:02.440 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b5427b66:::benchmark_data_smithi031_454752_object113:head# 2024-04-25T09:18:02.440 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b59f1134:::benchmark_data_smithi031_454752_object301:head# 2024-04-25T09:18:02.440 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b66cf944:::benchmark_data_smithi031_460268_object183:head# 2024-04-25T09:18:02.441 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b66ddae8:::benchmark_data_smithi031_454752_object206:head# 2024-04-25T09:18:02.442 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b6a31cad:::benchmark_data_smithi031_454752_object27:head# 2024-04-25T09:18:02.442 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b7073e4b:::benchmark_data_smithi031_454752_object126:head# 2024-04-25T09:18:02.443 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b713a907:::benchmark_data_smithi031_460268_object74:head# 2024-04-25T09:18:02.443 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b82823fa:::benchmark_data_smithi031_460268_object192:head# 2024-04-25T09:18:02.444 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b83f72b1:::benchmark_data_smithi031_454752_object162:head# 2024-04-25T09:18:02.444 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:18:02.445 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b8e53c75:::benchmark_data_smithi031_460268_object50:head# 2024-04-25T09:18:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b94a5f10:::benchmark_data_smithi031_454752_object348:head# 2024-04-25T09:18:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b9a0d9e0:::benchmark_data_smithi031_454752_object342:head# 2024-04-25T09:18:02.446 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b9e6e194:::benchmark_data_smithi031_460268_object34:head# 2024-04-25T09:18:02.447 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb052c00:::benchmark_data_smithi031_454752_object88:head# 2024-04-25T09:18:02.447 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb48e6d2:::benchmark_data_smithi031_454752_object241:head# 2024-04-25T09:18:02.448 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb57b519:::benchmark_data_smithi031_454752_object50:head# 2024-04-25T09:18:02.448 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb6af9e7:::benchmark_data_smithi031_454752_object373:head# 2024-04-25T09:18:02.449 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bbf3767d:::benchmark_data_smithi031_460268_object161:head# 2024-04-25T09:18:02.449 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bc520c76:::benchmark_data_smithi031_460268_object105:head# 2024-04-25T09:18:02.450 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bc8ac351:::benchmark_data_smithi031_460268_object36:head# 2024-04-25T09:18:02.450 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bca0f028:::benchmark_data_smithi031_454752_object20:head# 2024-04-25T09:18:02.451 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bd00cd0f:::benchmark_data_smithi031_454752_object24:head# 2024-04-25T09:18:02.451 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bd3a993f:::benchmark_data_smithi031_454752_object255:head# 2024-04-25T09:18:02.452 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bd3fca90:::benchmark_data_smithi031_460268_object128:head# 2024-04-25T09:18:02.452 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bdd858f4:::benchmark_data_smithi031_454752_object228:head# 2024-04-25T09:18:02.453 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bdece741:::benchmark_data_smithi031_460268_object150:head# 2024-04-25T09:18:02.453 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bf17b80d:::benchmark_data_smithi031_460268_object101:head# 2024-04-25T09:18:02.453 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bfa77210:::benchmark_data_smithi031_454752_object67:head# 2024-04-25T09:18:02.454 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c0619f0c:::benchmark_data_smithi031_454752_object247:head# 2024-04-25T09:18:02.455 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c0b886c3:::benchmark_data_smithi031_454752_object25:head# 2024-04-25T09:18:02.455 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c1540ecc:::benchmark_data_smithi031_460268_object4:head# 2024-04-25T09:18:02.456 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c1bc9d65:::benchmark_data_smithi031_460268_object38:head# 2024-04-25T09:18:02.456 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c217c5ee:::benchmark_data_smithi031_460268_object115:head# 2024-04-25T09:18:02.456 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c229294b:::benchmark_data_smithi031_454752_object127:head# 2024-04-25T09:18:02.457 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c2647231:::benchmark_data_smithi031_460268_object174:head# 2024-04-25T09:18:02.457 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c2f8ad7b:::benchmark_data_smithi031_454752_object104:head# 2024-04-25T09:18:02.459 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c4d896cc:::benchmark_data_smithi031_460268_object153:head# 2024-04-25T09:18:02.459 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c4fe132d:::benchmark_data_smithi031_454752_object152:head# 2024-04-25T09:18:02.459 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c5aa9351:::benchmark_data_smithi031_460268_object119:head# 2024-04-25T09:18:02.459 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c5dc4341:::benchmark_data_smithi031_454752_object166:head# 2024-04-25T09:18:02.460 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c6367955:::benchmark_data_smithi031_454752_object144:head# 2024-04-25T09:18:02.460 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c664156b:::benchmark_data_smithi031_454752_object238:head# 2024-04-25T09:18:02.461 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c674ecb3:::benchmark_data_smithi031_460268_object117:head# 2024-04-25T09:18:02.462 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c78167ae:::benchmark_data_smithi031_454752_object85:head# 2024-04-25T09:18:02.462 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c79d3970:::benchmark_data_smithi031_454752_object221:head# 2024-04-25T09:18:02.463 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c86d58be:::benchmark_data_smithi031_460268_object65:head# 2024-04-25T09:18:02.463 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c8eb0827:::benchmark_data_smithi031_454752_object298:head# 2024-04-25T09:18:02.463 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c96d11d1:::benchmark_data_smithi031_460268_object30:head# 2024-04-25T09:18:02.464 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca1d51a2:::benchmark_data_smithi031_454752_object280:head# 2024-04-25T09:18:02.464 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca1f007e:::benchmark_data_smithi031_460268_object57:head# 2024-04-25T09:18:02.465 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cab67049:::benchmark_data_smithi031_454752_object91:head# 2024-04-25T09:18:02.465 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cad688fc:::benchmark_data_smithi031_454752_object351:head# 2024-04-25T09:18:02.465 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:caf98509:::benchmark_data_smithi031_454752_object196:head# 2024-04-25T09:18:02.466 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cc58c178:::benchmark_data_smithi031_454752_object32:head# 2024-04-25T09:18:02.466 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cc9c514e:::benchmark_data_smithi031_454752_object65:head# 2024-04-25T09:18:02.467 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cca29ffb:::benchmark_data_smithi031_454752_object80:head# 2024-04-25T09:18:02.468 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ccbd2f1f:::benchmark_data_smithi031_454752_object184:head# 2024-04-25T09:18:02.468 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ccc8b271:::benchmark_data_smithi031_460268_object33:head# 2024-04-25T09:18:02.469 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cce211e5:::benchmark_data_smithi031_460268_object42:head# 2024-04-25T09:18:02.469 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cd8693fa:::benchmark_data_smithi031_454752_object398:head# 2024-04-25T09:18:02.469 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ce45c38a:::benchmark_data_smithi031_460268_object93:head# 2024-04-25T09:18:02.470 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ceb870b0:::benchmark_data_smithi031_454752_object222:head# 2024-04-25T09:18:02.471 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d030878f:::benchmark_data_smithi031_460268_object45:head# 2024-04-25T09:18:02.471 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d093eb2e:::benchmark_data_smithi031_454752_object311:head# 2024-04-25T09:18:02.472 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d231d661:::benchmark_data_smithi031_454752_object177:head# 2024-04-25T09:18:02.472 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d2ae4d7f:::benchmark_data_smithi031_454752_object51:head# 2024-04-25T09:18:02.472 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d35c9cea:::benchmark_data_smithi031_454752_object317:head# 2024-04-25T09:18:02.473 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d40521f8:::benchmark_data_smithi031_460268_object52:head# 2024-04-25T09:18:02.473 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d4438560:::benchmark_data_smithi031_454752_object143:head# 2024-04-25T09:18:02.474 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d452fc99:::benchmark_data_smithi031_454752_object45:head# 2024-04-25T09:18:02.474 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d6165b5f:::benchmark_data_smithi031_460268_object7:head# 2024-04-25T09:18:02.475 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d6574e4a:::benchmark_data_smithi031_454752_object60:head# 2024-04-25T09:18:02.475 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d73435c2:::benchmark_data_smithi031_454752_object118:head# 2024-04-25T09:18:02.475 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d75560e8:::benchmark_data_smithi031_454752_object128:head# 2024-04-25T09:18:02.478 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d75a04bb:::benchmark_data_smithi031_454752_object98:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d75b7183:::benchmark_data_smithi031_454752_object230:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d7bf5e5b:::benchmark_data_smithi031_454752_object239:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d824c896:::benchmark_data_smithi031_454752_object286:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:da211111:::benchmark_data_smithi031_454752_object159:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:da25a4ff:::benchmark_data_smithi031_454752_object75:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:da449e8d:::benchmark_data_smithi031_460268_object91:head# 2024-04-25T09:18:02.479 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:db18fece:::benchmark_data_smithi031_454752_object250:head# 2024-04-25T09:18:02.480 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dc4325f0:::benchmark_data_smithi031_454752_object131:head# 2024-04-25T09:18:02.481 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd2df7d0:::benchmark_data_smithi031_460268_object20:head# 2024-04-25T09:18:02.481 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd89516f:::benchmark_data_smithi031_460268_object112:head# 2024-04-25T09:18:02.481 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dda1d8b9:::benchmark_data_smithi031_460268_object28:head# 2024-04-25T09:18:02.482 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:deb7d977:::benchmark_data_smithi031_460268_object134:head# 2024-04-25T09:18:02.482 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dedcc3de:::benchmark_data_smithi031_454752_object148:head# 2024-04-25T09:18:02.483 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:deec6ff8:::benchmark_data_smithi031_454752_object224:head# 2024-04-25T09:18:02.484 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e0881e69:::benchmark_data_smithi031_454752_object293:head# 2024-04-25T09:18:02.484 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e1341170:::benchmark_data_smithi031_454752_object132:head# 2024-04-25T09:18:02.485 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e17ed360:::benchmark_data_smithi031_454752_object140:head# 2024-04-25T09:18:02.485 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e3835ce2:::benchmark_data_smithi031_460268_object188:head# 2024-04-25T09:18:02.486 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e3be378c:::benchmark_data_smithi031_454752_object188:head# 2024-04-25T09:18:02.486 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e3c37152:::benchmark_data_smithi031_454752_object155:head# 2024-04-25T09:18:02.487 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e431150e:::benchmark_data_smithi031_454752_object309:head# 2024-04-25T09:18:02.487 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e44b8a2c:::benchmark_data_smithi031_454752_object146:head# 2024-04-25T09:18:02.488 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e469a838:::benchmark_data_smithi031_460268_object63:head# 2024-04-25T09:18:02.488 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e4a3dd60:::benchmark_data_smithi031_454752_object341:head# 2024-04-25T09:18:02.489 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e53228e8:::benchmark_data_smithi031_454752_object364:head# 2024-04-25T09:18:02.490 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e53430a0:::benchmark_data_smithi031_454752_object336:head# 2024-04-25T09:18:02.490 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e5497360:::benchmark_data_smithi031_454752_object141:head# 2024-04-25T09:18:02.491 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e5ae028f:::benchmark_data_smithi031_454752_object254:head# 2024-04-25T09:18:02.491 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e653ea24:::benchmark_data_smithi031_460268_object51:head# 2024-04-25T09:18:02.492 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e72dd955:::benchmark_data_smithi031_454752_object105:head# 2024-04-25T09:18:02.493 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e75527e1:::benchmark_data_smithi031_454752_object365:head# 2024-04-25T09:18:02.493 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e7710ac5:::benchmark_data_smithi031_460268_object53:head# 2024-04-25T09:18:02.494 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e8ddccdd:::benchmark_data_smithi031_460268_object31:head# 2024-04-25T09:18:02.495 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e8e4cd11:::benchmark_data_smithi031_454752_object171:head# 2024-04-25T09:18:02.495 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e94a0572:::benchmark_data_smithi031_454752_object6:head# 2024-04-25T09:18:02.495 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e991363a:::benchmark_data_smithi031_454752_object52:head# 2024-04-25T09:18:02.496 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e9cfc042:::benchmark_data_smithi031_460268_object24:head# 2024-04-25T09:18:02.497 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ea07a4ce:::benchmark_data_smithi031_460268_object3:head# 2024-04-25T09:18:02.497 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ea70172b:::benchmark_data_smithi031_454752_object191:head# 2024-04-25T09:18:02.497 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:eaf3d89c:::benchmark_data_smithi031_454752_object358:head# 2024-04-25T09:18:02.498 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ec070df6:::benchmark_data_smithi031_454752_object234:head# 2024-04-25T09:18:02.498 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ec82b70c:::benchmark_data_smithi031_454752_object246:head# 2024-04-25T09:18:02.499 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ed17f3a5:::benchmark_data_smithi031_454752_object389:head# 2024-04-25T09:18:02.499 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:edd73bf9:::benchmark_data_smithi031_454752_object95:head# 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ee0f64f0:::benchmark_data_smithi031_454752_object49:head# 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ee2ac15d:::benchmark_data_smithi031_460268_object100:head# 2024-04-25T09:18:02.501 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f054ad11:::benchmark_data_smithi031_454752_object145:head# 2024-04-25T09:18:02.502 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f0cae848:::benchmark_data_smithi031_460268_object136:head# 2024-04-25T09:18:02.502 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f102181e:::benchmark_data_smithi031_454752_object288:head# 2024-04-25T09:18:02.503 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f155a6c1:::benchmark_data_smithi031_460268_object124:head# 2024-04-25T09:18:02.503 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f21d79de:::benchmark_data_smithi031_454752_object66:head# 2024-04-25T09:18:02.504 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f2b747ee:::benchmark_data_smithi031_454752_object314:head# 2024-04-25T09:18:02.505 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f308d551:::benchmark_data_smithi031_460268_object84:head# 2024-04-25T09:18:02.505 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f34b8ad2:::benchmark_data_smithi031_454752_object240:head# 2024-04-25T09:18:02.506 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f4863048:::benchmark_data_smithi031_454752_object349:head# 2024-04-25T09:18:02.506 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f4ab8147:::benchmark_data_smithi031_454752_object337:head# 2024-04-25T09:18:02.507 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f53a131c:::benchmark_data_smithi031_460268_object17:head# 2024-04-25T09:18:02.507 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f544082b:::benchmark_data_smithi031_460268_object81:head# 2024-04-25T09:18:02.508 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f547a4d2:::benchmark_data_smithi031_454752_object369:head# 2024-04-25T09:18:02.508 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f5b63e7c:::benchmark_data_smithi031_460268_object171:head# 2024-04-25T09:18:02.509 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f6236fee:::benchmark_data_smithi031_454752_object289:head# 2024-04-25T09:18:02.509 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f635f136:::benchmark_data_smithi031_454752_object23:head# 2024-04-25T09:18:02.510 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f6e629fe:::benchmark_data_smithi031_454752_object57:head# 2024-04-25T09:18:02.510 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f70fb177:::benchmark_data_smithi031_454752_object265:head# 2024-04-25T09:18:02.511 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f77401b8:::benchmark_data_smithi031_454752_object71:head# 2024-04-25T09:18:02.511 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f7954077:::benchmark_data_smithi031_460268_object145:head# 2024-04-25T09:18:02.512 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f870ff0e:::benchmark_data_smithi031_454752_object312:head# 2024-04-25T09:18:02.513 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f8855049:::benchmark_data_smithi031_460268_object94:head# 2024-04-25T09:18:02.513 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f90eb4af:::benchmark_data_smithi031_460268_object185:head# 2024-04-25T09:18:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f93bf106:::benchmark_data_smithi031_460268_object72:head# 2024-04-25T09:18:02.514 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fa3f28f3:::benchmark_data_smithi031_460268_object166:head# 2024-04-25T09:18:02.515 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:faefa441:::benchmark_data_smithi031_454752_object269:head# 2024-04-25T09:18:02.516 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb1204b7:::benchmark_data_smithi031_454752_object331:head# 2024-04-25T09:18:02.516 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb3355e2:::benchmark_data_smithi031_454752_object363:head# 2024-04-25T09:18:02.517 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb5c543a:::benchmark_data_smithi031_454752_object329:head# 2024-04-25T09:18:02.517 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fbfbcb4a:::benchmark_data_smithi031_460268_object104:head# 2024-04-25T09:18:02.518 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fcf4159f:::benchmark_data_smithi031_454752_object185:head# 2024-04-25T09:18:02.518 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fe5d917d:::benchmark_data_smithi031_460268_object182:head# 2024-04-25T09:18:02.519 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fe7c790a:::benchmark_data_smithi031_454752_object16:head# 2024-04-25T09:18:02.519 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ffeaaa0c:::benchmark_data_smithi031_454752_object139:head# 2024-04-25T09:18:02.520 INFO:tasks.workunit.client.0.smithi031.stderr:Export successful 2024-04-25T09:18:02.520 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:18:02.520 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:18:02.520 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:18:03.410 INFO:tasks.workunit.client.0.smithi031.stderr:Remove successful 2024-04-25T09:18:03.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:18:04.754 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:18:04.754 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.1 2024-04-25T09:18:04.755 INFO:tasks.workunit.client.0.smithi031.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:18:04.755 INFO:tasks.workunit.client.0.smithi031.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:18:05.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:18:06.160 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:18:06.160 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T09:18:06.161 INFO:tasks.workunit.client.0.smithi031.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:18:06.161 INFO:tasks.workunit.client.0.smithi031.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:18:06.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:18:07.346 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:18:07.346 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.1 2024-04-25T09:18:07.346 INFO:tasks.workunit.client.0.smithi031.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:18:07.347 INFO:tasks.workunit.client.0.smithi031.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:18:07.347 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:825fce9d:::benchmark_data_smithi031_454752_object9:head# 2024-04-25T09:18:07.347 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.408 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:82fe0341:::benchmark_data_smithi031_454752_object167:head# 2024-04-25T09:18:07.409 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.442 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:830b3926:::benchmark_data_smithi031_454752_object175:head# 2024-04-25T09:18:07.442 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.475 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:839ff2c1:::benchmark_data_smithi031_454752_object178:head# 2024-04-25T09:18:07.476 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.509 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:83bed9ab:::benchmark_data_smithi031_454752_object192:head# 2024-04-25T09:18:07.509 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.542 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:84bae7de:::benchmark_data_smithi031_454752_object19:head# 2024-04-25T09:18:07.542 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.576 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8514352c:::benchmark_data_smithi031_454752_object84:head# 2024-04-25T09:18:07.576 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.609 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:85e422cb:::benchmark_data_smithi031_454752_object183:head# 2024-04-25T09:18:07.609 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.642 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8696227e:::benchmark_data_smithi031_454752_object304:head# 2024-04-25T09:18:07.643 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.676 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:86aa36c1:::benchmark_data_smithi031_454752_object179:head# 2024-04-25T09:18:07.676 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.709 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:878ba262:::benchmark_data_smithi031_454752_object281:head# 2024-04-25T09:18:07.710 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.743 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:891fd6e5:::benchmark_data_smithi031_454752_object28:head# 2024-04-25T09:18:07.743 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.776 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8941388a:::benchmark_data_smithi031_454752_object87:head# 2024-04-25T09:18:07.776 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.810 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8a4773c4:::benchmark_data_smithi031_454752_object103:head# 2024-04-25T09:18:07.810 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.843 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8abbf253:::benchmark_data_smithi031_454752_object97:head# 2024-04-25T09:18:07.843 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.877 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8adf42cb:::benchmark_data_smithi031_454752_object180:head# 2024-04-25T09:18:07.877 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.910 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8ae888df:::benchmark_data_smithi031_454752_object382:head# 2024-04-25T09:18:07.910 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.943 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8b291374:::benchmark_data_smithi031_454752_object273:head# 2024-04-25T09:18:07.944 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8b621052:::benchmark_data_smithi031_454752_object368:head# 2024-04-25T09:18:07.977 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.010 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8c90b5b1:::benchmark_data_smithi031_454752_object161:head# 2024-04-25T09:18:08.011 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.044 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8d29fb8d:::benchmark_data_smithi031_454752_object327:head# 2024-04-25T09:18:08.044 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.077 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8db4e958:::benchmark_data_smithi031_454752_object385:head# 2024-04-25T09:18:08.077 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.111 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8fa26b27:::benchmark_data_smithi031_454752_object256:head# 2024-04-25T09:18:08.111 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.144 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:90588a1a:::benchmark_data_smithi031_454752_object378:head# 2024-04-25T09:18:08.144 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.178 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:907e8918:::benchmark_data_smithi031_454752_object266:head# 2024-04-25T09:18:08.178 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.212 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9277f059:::benchmark_data_smithi031_454752_object2:head# 2024-04-25T09:18:08.212 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.246 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:946e6356:::benchmark_data_smithi031_454752_object338:head# 2024-04-25T09:18:08.246 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.278 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9586170f:::benchmark_data_smithi031_454752_object13:head# 2024-04-25T09:18:08.278 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.311 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:95c08ad2:::benchmark_data_smithi031_454752_object210:head# 2024-04-25T09:18:08.312 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.353 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:95cd8cdf:::benchmark_data_smithi031_454752_object383:head# 2024-04-25T09:18:08.353 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.387 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9609476c:::benchmark_data_smithi031_454752_object114:head# 2024-04-25T09:18:08.387 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.420 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:967bc283:::benchmark_data_smithi031_454752_object231:head# 2024-04-25T09:18:08.420 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.454 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96801ae8:::benchmark_data_smithi031_454752_object207:head# 2024-04-25T09:18:08.454 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.487 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96a964d2:::benchmark_data_smithi031_454752_object38:head# 2024-04-25T09:18:08.487 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.520 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96aa787d:::benchmark_data_smithi031_454752_object322:head# 2024-04-25T09:18:08.521 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.554 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96d63edc:::benchmark_data_smithi031_454752_object352:head# 2024-04-25T09:18:08.554 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.596 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:98e8b5bf:::benchmark_data_smithi031_454752_object216:head# 2024-04-25T09:18:08.597 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.646 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:99e3add0:::benchmark_data_smithi031_454752_object5:head# 2024-04-25T09:18:08.646 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.688 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a07c74e:::benchmark_data_smithi031_454752_object252:head# 2024-04-25T09:18:08.688 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.755 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a72fd5f:::benchmark_data_smithi031_454752_object297:head# 2024-04-25T09:18:08.755 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.841 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a9f4dbd:::benchmark_data_smithi031_454752_object321:head# 2024-04-25T09:18:08.841 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.924 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9b86cf51:::benchmark_data_smithi031_454752_object225:head# 2024-04-25T09:18:08.924 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.965 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9be87125:::benchmark_data_smithi031_454752_object116:head# 2024-04-25T09:18:08.965 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:08.998 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9bf0e6c9:::benchmark_data_smithi031_454752_object47:head# 2024-04-25T09:18:08.998 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.032 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9c836d36:::benchmark_data_smithi031_454752_object44:head# 2024-04-25T09:18:09.032 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.073 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9cb1d2e2:::benchmark_data_smithi031_454752_object362:head# 2024-04-25T09:18:09.074 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.132 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9cbe41c2:::benchmark_data_smithi031_454752_object374:head# 2024-04-25T09:18:09.132 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.191 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9ce31ece:::benchmark_data_smithi031_454752_object308:head# 2024-04-25T09:18:09.191 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.224 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9df3f789:::benchmark_data_smithi031_454752_object394:head# 2024-04-25T09:18:09.224 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.258 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9e952085:::benchmark_data_smithi031_454752_object154:head# 2024-04-25T09:18:09.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.291 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9f2a7ece:::benchmark_data_smithi031_454752_object251:head# 2024-04-25T09:18:09.291 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.326 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a114ad36:::benchmark_data_smithi031_454752_object70:head# 2024-04-25T09:18:09.326 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.358 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a419031b:::benchmark_data_smithi031_454752_object81:head# 2024-04-25T09:18:09.358 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.391 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a45f5752:::benchmark_data_smithi031_454752_object39:head# 2024-04-25T09:18:09.392 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.425 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a46e3cab:::benchmark_data_smithi031_454752_object193:head# 2024-04-25T09:18:09.425 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.458 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a46e59f0:::benchmark_data_smithi031_454752_object130:head# 2024-04-25T09:18:09.458 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.492 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a75374ab:::benchmark_data_smithi031_454752_object190:head# 2024-04-25T09:18:09.492 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.525 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8271a67:::benchmark_data_smithi031_454752_object33:head# 2024-04-25T09:18:09.525 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.559 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a853f81a:::benchmark_data_smithi031_454752_object379:head# 2024-04-25T09:18:09.559 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.592 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8c6bb46:::benchmark_data_smithi031_454752_object72:head# 2024-04-25T09:18:09.592 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.626 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a942e3ba:::benchmark_data_smithi031_454752_object74:head# 2024-04-25T09:18:09.626 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.659 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a98c4d7c:::benchmark_data_smithi031_454752_object37:head# 2024-04-25T09:18:09.659 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.692 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9e307d2:::benchmark_data_smithi031_454752_object318:head# 2024-04-25T09:18:09.693 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.726 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aab383a6:::benchmark_data_smithi031_454752_object112:head# 2024-04-25T09:18:09.726 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.759 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aaf58300:::benchmark_data_smithi031_454752_object208:head# 2024-04-25T09:18:09.759 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.801 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab0f7dc2:::benchmark_data_smithi031_454752_object119:head# 2024-04-25T09:18:09.801 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.835 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab1cf81e:::benchmark_data_smithi031_454752_object263:head# 2024-04-25T09:18:09.835 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.868 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab49749c:::benchmark_data_smithi031_454752_object359:head# 2024-04-25T09:18:09.868 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.901 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ac1d5d1f:::benchmark_data_smithi031_454752_object135:head# 2024-04-25T09:18:09.902 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.935 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ac7ea5bf:::benchmark_data_smithi031_454752_object217:head# 2024-04-25T09:18:09.935 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:09.968 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:acc3e569:::benchmark_data_smithi031_454752_object290:head# 2024-04-25T09:18:09.969 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.002 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ad398d0f:::benchmark_data_smithi031_454752_object125:head# 2024-04-25T09:18:10.002 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.036 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ae91e1ef:::benchmark_data_smithi031_454752_object115:head# 2024-04-25T09:18:10.036 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.069 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aea3dfa5:::benchmark_data_smithi031_454752_object388:head# 2024-04-25T09:18:10.069 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.102 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aec2dd45:::benchmark_data_smithi031_454752_object275:head# 2024-04-25T09:18:10.102 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.136 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af668e34:::benchmark_data_smithi031_454752_object302:head# 2024-04-25T09:18:10.136 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.169 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af9a9573:::benchmark_data_smithi031_454752_object58:head# 2024-04-25T09:18:10.169 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.203 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b089bb4f:::benchmark_data_smithi031_454752_object218:head# 2024-04-25T09:18:10.203 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.236 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b1674de8:::benchmark_data_smithi031_454752_object194:head# 2024-04-25T09:18:10.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.270 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b2930ee9:::benchmark_data_smithi031_454752_object73:head# 2024-04-25T09:18:10.270 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.303 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b329d82c:::benchmark_data_smithi031_454752_object200:head# 2024-04-25T09:18:10.303 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.336 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b3c75192:::benchmark_data_smithi031_454752_object325:head# 2024-04-25T09:18:10.336 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.370 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b45e9911:::benchmark_data_smithi031_454752_object53:head# 2024-04-25T09:18:10.370 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.403 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b4f7e75f:::benchmark_data_smithi031_454752_object296:head# 2024-04-25T09:18:10.403 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b51eab52:::benchmark_data_smithi031_454752_object393:head# 2024-04-25T09:18:10.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b54079d2:::benchmark_data_smithi031_454752_object319:head# 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.504 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b5427b66:::benchmark_data_smithi031_454752_object113:head# 2024-04-25T09:18:10.504 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.558 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b59f1134:::benchmark_data_smithi031_454752_object301:head# 2024-04-25T09:18:10.558 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.604 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b66ddae8:::benchmark_data_smithi031_454752_object206:head# 2024-04-25T09:18:10.604 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.638 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b6a31cad:::benchmark_data_smithi031_454752_object27:head# 2024-04-25T09:18:10.638 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.671 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b7073e4b:::benchmark_data_smithi031_454752_object126:head# 2024-04-25T09:18:10.671 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b83f72b1:::benchmark_data_smithi031_454752_object162:head# 2024-04-25T09:18:10.704 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:18:10.738 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.771 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b94a5f10:::benchmark_data_smithi031_454752_object348:head# 2024-04-25T09:18:10.771 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.805 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b9a0d9e0:::benchmark_data_smithi031_454752_object342:head# 2024-04-25T09:18:10.805 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.849 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb052c00:::benchmark_data_smithi031_454752_object88:head# 2024-04-25T09:18:10.850 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.897 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb48e6d2:::benchmark_data_smithi031_454752_object241:head# 2024-04-25T09:18:10.897 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.939 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb57b519:::benchmark_data_smithi031_454752_object50:head# 2024-04-25T09:18:10.939 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:10.972 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb6af9e7:::benchmark_data_smithi031_454752_object373:head# 2024-04-25T09:18:10.972 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.006 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bca0f028:::benchmark_data_smithi031_454752_object20:head# 2024-04-25T09:18:11.006 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.039 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd00cd0f:::benchmark_data_smithi031_454752_object24:head# 2024-04-25T09:18:11.040 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.074 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd3a993f:::benchmark_data_smithi031_454752_object255:head# 2024-04-25T09:18:11.075 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.149 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bdd858f4:::benchmark_data_smithi031_454752_object228:head# 2024-04-25T09:18:11.150 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.215 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bfa77210:::benchmark_data_smithi031_454752_object67:head# 2024-04-25T09:18:11.215 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.240 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0619f0c:::benchmark_data_smithi031_454752_object247:head# 2024-04-25T09:18:11.241 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.266 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0b886c3:::benchmark_data_smithi031_454752_object25:head# 2024-04-25T09:18:11.266 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.291 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c229294b:::benchmark_data_smithi031_454752_object127:head# 2024-04-25T09:18:11.291 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.316 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c2f8ad7b:::benchmark_data_smithi031_454752_object104:head# 2024-04-25T09:18:11.316 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.349 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c4fe132d:::benchmark_data_smithi031_454752_object152:head# 2024-04-25T09:18:11.349 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.383 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c5dc4341:::benchmark_data_smithi031_454752_object166:head# 2024-04-25T09:18:11.383 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.416 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c6367955:::benchmark_data_smithi031_454752_object144:head# 2024-04-25T09:18:11.416 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.449 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c664156b:::benchmark_data_smithi031_454752_object238:head# 2024-04-25T09:18:11.449 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.483 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c78167ae:::benchmark_data_smithi031_454752_object85:head# 2024-04-25T09:18:11.483 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.516 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c79d3970:::benchmark_data_smithi031_454752_object221:head# 2024-04-25T09:18:11.516 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.549 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c8eb0827:::benchmark_data_smithi031_454752_object298:head# 2024-04-25T09:18:11.550 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.583 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca1d51a2:::benchmark_data_smithi031_454752_object280:head# 2024-04-25T09:18:11.583 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.617 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cab67049:::benchmark_data_smithi031_454752_object91:head# 2024-04-25T09:18:11.617 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.650 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cad688fc:::benchmark_data_smithi031_454752_object351:head# 2024-04-25T09:18:11.650 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.683 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:caf98509:::benchmark_data_smithi031_454752_object196:head# 2024-04-25T09:18:11.684 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.717 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc58c178:::benchmark_data_smithi031_454752_object32:head# 2024-04-25T09:18:11.717 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.750 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc9c514e:::benchmark_data_smithi031_454752_object65:head# 2024-04-25T09:18:11.750 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.784 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cca29ffb:::benchmark_data_smithi031_454752_object80:head# 2024-04-25T09:18:11.784 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.817 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ccbd2f1f:::benchmark_data_smithi031_454752_object184:head# 2024-04-25T09:18:11.817 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.851 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cd8693fa:::benchmark_data_smithi031_454752_object398:head# 2024-04-25T09:18:11.851 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.884 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ceb870b0:::benchmark_data_smithi031_454752_object222:head# 2024-04-25T09:18:11.884 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.918 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d093eb2e:::benchmark_data_smithi031_454752_object311:head# 2024-04-25T09:18:11.918 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.951 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d231d661:::benchmark_data_smithi031_454752_object177:head# 2024-04-25T09:18:11.951 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:11.984 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d2ae4d7f:::benchmark_data_smithi031_454752_object51:head# 2024-04-25T09:18:11.985 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.018 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d35c9cea:::benchmark_data_smithi031_454752_object317:head# 2024-04-25T09:18:12.018 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.093 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d4438560:::benchmark_data_smithi031_454752_object143:head# 2024-04-25T09:18:12.093 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.127 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d452fc99:::benchmark_data_smithi031_454752_object45:head# 2024-04-25T09:18:12.129 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.160 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d6574e4a:::benchmark_data_smithi031_454752_object60:head# 2024-04-25T09:18:12.161 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.193 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d73435c2:::benchmark_data_smithi031_454752_object118:head# 2024-04-25T09:18:12.194 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.227 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75560e8:::benchmark_data_smithi031_454752_object128:head# 2024-04-25T09:18:12.227 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.260 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75a04bb:::benchmark_data_smithi031_454752_object98:head# 2024-04-25T09:18:12.261 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.294 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75b7183:::benchmark_data_smithi031_454752_object230:head# 2024-04-25T09:18:12.294 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.327 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d7bf5e5b:::benchmark_data_smithi031_454752_object239:head# 2024-04-25T09:18:12.328 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.361 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d824c896:::benchmark_data_smithi031_454752_object286:head# 2024-04-25T09:18:12.361 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.394 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da211111:::benchmark_data_smithi031_454752_object159:head# 2024-04-25T09:18:12.394 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.428 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da25a4ff:::benchmark_data_smithi031_454752_object75:head# 2024-04-25T09:18:12.428 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.461 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:db18fece:::benchmark_data_smithi031_454752_object250:head# 2024-04-25T09:18:12.461 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.495 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dc4325f0:::benchmark_data_smithi031_454752_object131:head# 2024-04-25T09:18:12.495 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.528 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dedcc3de:::benchmark_data_smithi031_454752_object148:head# 2024-04-25T09:18:12.528 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.561 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:deec6ff8:::benchmark_data_smithi031_454752_object224:head# 2024-04-25T09:18:12.561 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.595 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e0881e69:::benchmark_data_smithi031_454752_object293:head# 2024-04-25T09:18:12.595 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.628 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e1341170:::benchmark_data_smithi031_454752_object132:head# 2024-04-25T09:18:12.629 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.662 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e17ed360:::benchmark_data_smithi031_454752_object140:head# 2024-04-25T09:18:12.662 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.695 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3be378c:::benchmark_data_smithi031_454752_object188:head# 2024-04-25T09:18:12.695 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.729 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3c37152:::benchmark_data_smithi031_454752_object155:head# 2024-04-25T09:18:12.729 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.762 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e431150e:::benchmark_data_smithi031_454752_object309:head# 2024-04-25T09:18:12.762 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.796 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e44b8a2c:::benchmark_data_smithi031_454752_object146:head# 2024-04-25T09:18:12.796 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.841 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e4a3dd60:::benchmark_data_smithi031_454752_object341:head# 2024-04-25T09:18:12.841 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.905 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e53228e8:::benchmark_data_smithi031_454752_object364:head# 2024-04-25T09:18:12.905 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.938 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e53430a0:::benchmark_data_smithi031_454752_object336:head# 2024-04-25T09:18:12.938 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:12.971 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e5497360:::benchmark_data_smithi031_454752_object141:head# 2024-04-25T09:18:12.972 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.005 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e5ae028f:::benchmark_data_smithi031_454752_object254:head# 2024-04-25T09:18:13.005 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.038 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e72dd955:::benchmark_data_smithi031_454752_object105:head# 2024-04-25T09:18:13.039 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.082 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e75527e1:::benchmark_data_smithi031_454752_object365:head# 2024-04-25T09:18:13.082 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.158 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e8e4cd11:::benchmark_data_smithi031_454752_object171:head# 2024-04-25T09:18:13.158 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.231 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e94a0572:::benchmark_data_smithi031_454752_object6:head# 2024-04-25T09:18:13.231 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.265 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e991363a:::benchmark_data_smithi031_454752_object52:head# 2024-04-25T09:18:13.265 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.298 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ea70172b:::benchmark_data_smithi031_454752_object191:head# 2024-04-25T09:18:13.298 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.331 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:eaf3d89c:::benchmark_data_smithi031_454752_object358:head# 2024-04-25T09:18:13.332 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.365 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ec070df6:::benchmark_data_smithi031_454752_object234:head# 2024-04-25T09:18:13.365 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.398 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ec82b70c:::benchmark_data_smithi031_454752_object246:head# 2024-04-25T09:18:13.399 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.432 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ed17f3a5:::benchmark_data_smithi031_454752_object389:head# 2024-04-25T09:18:13.432 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.457 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:edd73bf9:::benchmark_data_smithi031_454752_object95:head# 2024-04-25T09:18:13.457 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.482 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ee0f64f0:::benchmark_data_smithi031_454752_object49:head# 2024-04-25T09:18:13.482 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.507 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f054ad11:::benchmark_data_smithi031_454752_object145:head# 2024-04-25T09:18:13.507 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.532 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f102181e:::benchmark_data_smithi031_454752_object288:head# 2024-04-25T09:18:13.532 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.557 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f21d79de:::benchmark_data_smithi031_454752_object66:head# 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.583 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f2b747ee:::benchmark_data_smithi031_454752_object314:head# 2024-04-25T09:18:13.583 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.608 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f34b8ad2:::benchmark_data_smithi031_454752_object240:head# 2024-04-25T09:18:13.608 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.641 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4863048:::benchmark_data_smithi031_454752_object349:head# 2024-04-25T09:18:13.641 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.719 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4ab8147:::benchmark_data_smithi031_454752_object337:head# 2024-04-25T09:18:13.719 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.783 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f547a4d2:::benchmark_data_smithi031_454752_object369:head# 2024-04-25T09:18:13.784 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.826 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6236fee:::benchmark_data_smithi031_454752_object289:head# 2024-04-25T09:18:13.826 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.901 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f635f136:::benchmark_data_smithi031_454752_object23:head# 2024-04-25T09:18:13.901 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.951 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6e629fe:::benchmark_data_smithi031_454752_object57:head# 2024-04-25T09:18:13.951 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:13.985 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f70fb177:::benchmark_data_smithi031_454752_object265:head# 2024-04-25T09:18:13.985 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.018 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f77401b8:::benchmark_data_smithi031_454752_object71:head# 2024-04-25T09:18:14.018 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.052 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f870ff0e:::benchmark_data_smithi031_454752_object312:head# 2024-04-25T09:18:14.052 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.085 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:faefa441:::benchmark_data_smithi031_454752_object269:head# 2024-04-25T09:18:14.085 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.119 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb1204b7:::benchmark_data_smithi031_454752_object331:head# 2024-04-25T09:18:14.119 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.152 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb3355e2:::benchmark_data_smithi031_454752_object363:head# 2024-04-25T09:18:14.152 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.185 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb5c543a:::benchmark_data_smithi031_454752_object329:head# 2024-04-25T09:18:14.186 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.219 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fcf4159f:::benchmark_data_smithi031_454752_object185:head# 2024-04-25T09:18:14.219 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.253 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fe7c790a:::benchmark_data_smithi031_454752_object16:head# 2024-04-25T09:18:14.253 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.286 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ffeaaa0c:::benchmark_data_smithi031_454752_object139:head# 2024-04-25T09:18:14.286 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:14.319 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 10 info 1.1( v 10'189 (0'0,10'189] local-lis/les=7/8 n=189 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:18:14.353 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T09:18:14.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:18:15.948 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:18:15.948 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T09:18:15.949 INFO:tasks.workunit.client.0.smithi031.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:18:15.949 INFO:tasks.workunit.client.0.smithi031.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:18:15.949 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0005830b:::benchmark_data_smithi031_454752_object169:head# 2024-04-25T09:18:15.949 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.057 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0007ff59:::benchmark_data_smithi031_454752_object384:head# 2024-04-25T09:18:16.057 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.107 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00530cad:::benchmark_data_smithi031_454752_object366:head# 2024-04-25T09:18:16.107 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.157 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:007182ad:::benchmark_data_smithi031_454752_object150:head# 2024-04-25T09:18:16.157 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.216 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00808219:::benchmark_data_smithi031_454752_object10:head# 2024-04-25T09:18:16.216 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.257 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00ccfbba:::benchmark_data_smithi031_454752_object355:head# 2024-04-25T09:18:16.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.308 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00d7eda8:::benchmark_data_smithi031_454752_object357:head# 2024-04-25T09:18:16.308 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.366 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:01faf234:::benchmark_data_smithi031_454752_object272:head# 2024-04-25T09:18:16.366 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.425 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:021d5fe5:::benchmark_data_smithi031_454752_object109:head# 2024-04-25T09:18:16.425 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.467 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:02a87b95:::benchmark_data_smithi031_454752_object26:head# 2024-04-25T09:18:16.467 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.517 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03119e67:::benchmark_data_smithi031_454752_object120:head# 2024-04-25T09:18:16.517 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.575 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03a19a01:::benchmark_data_smithi031_454752_object375:head# 2024-04-25T09:18:16.576 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.634 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03d8d0d6:::benchmark_data_smithi031_454752_object307:head# 2024-04-25T09:18:16.634 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.693 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0407316c:::benchmark_data_smithi031_454752_object248:head# 2024-04-25T09:18:16.693 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.743 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:069ac774:::benchmark_data_smithi031_454752_object270:head# 2024-04-25T09:18:16.743 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.793 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:076ac3b3:::benchmark_data_smithi031_454752_object168:head# 2024-04-25T09:18:16.793 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.827 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:078ca8f4:::benchmark_data_smithi031_454752_object271:head# 2024-04-25T09:18:16.827 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.944 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:089737ee:::benchmark_data_smithi031_454752_object262:head# 2024-04-25T09:18:16.944 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:16.994 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0954f670:::benchmark_data_smithi031_454752_object223:head# 2024-04-25T09:18:16.994 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.057 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:09d319a2:::benchmark_data_smithi031_454752_object283:head# 2024-04-25T09:18:17.057 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.180 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a33ee0a:::benchmark_data_smithi031_454752_object107:head# 2024-04-25T09:18:17.180 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a588e5f:::benchmark_data_smithi031_454752_object305:head# 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.320 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ae0c865:::benchmark_data_smithi031_454752_object41:head# 2024-04-25T09:18:17.321 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.371 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0c38e0a3:::benchmark_data_smithi031_454752_object387:head# 2024-04-25T09:18:17.371 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.421 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0c4decad:::benchmark_data_smithi031_454752_object151:head# 2024-04-25T09:18:17.421 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.471 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0cc7bcb7:::benchmark_data_smithi031_454752_object333:head# 2024-04-25T09:18:17.471 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.538 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ce7b7bc:::benchmark_data_smithi031_454752_object353:head# 2024-04-25T09:18:17.539 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.605 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0d6e3436:::benchmark_data_smithi031_454752_object205:head# 2024-04-25T09:18:17.605 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.656 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0dd158a0:::benchmark_data_smithi031_454752_object76:head# 2024-04-25T09:18:17.656 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.706 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0df905cb:::benchmark_data_smithi031_454752_object181:head# 2024-04-25T09:18:17.706 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.756 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0f047668:::benchmark_data_smithi031_454752_object12:head# 2024-04-25T09:18:17.756 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.789 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:100b6e67:::benchmark_data_smithi031_454752_object122:head# 2024-04-25T09:18:17.790 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.823 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:105e9b29:::benchmark_data_smithi031_454752_object295:head# 2024-04-25T09:18:17.823 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.856 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:10ca342c:::benchmark_data_smithi031_454752_object203:head# 2024-04-25T09:18:17.856 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.890 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:11985b73:::benchmark_data_smithi031_454752_object294:head# 2024-04-25T09:18:17.890 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.923 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1215ebb0:::benchmark_data_smithi031_454752_object96:head# 2024-04-25T09:18:17.924 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.957 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:123f25d6:::benchmark_data_smithi031_454752_object339:head# 2024-04-25T09:18:17.957 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:17.991 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:12648b62:::benchmark_data_smithi031_454752_object235:head# 2024-04-25T09:18:17.991 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.024 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:13b5933b:::benchmark_data_smithi031_454752_object258:head# 2024-04-25T09:18:18.024 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.057 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:14a4f03a:::benchmark_data_smithi031_454752_object90:head# 2024-04-25T09:18:18.057 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.091 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:152756a2:::benchmark_data_smithi031_454752_object78:head# 2024-04-25T09:18:18.091 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.124 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17685eef:::benchmark_data_smithi031_454752_object35:head# 2024-04-25T09:18:18.124 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.157 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17d78309:::benchmark_data_smithi031_454752_object197:head# 2024-04-25T09:18:18.158 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.191 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:186323b7:::benchmark_data_smithi031_454752_object264:head# 2024-04-25T09:18:18.191 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.247 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18a96ee0:::benchmark_data_smithi031_454752_object340:head# 2024-04-25T09:18:18.247 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.283 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18abc847:::benchmark_data_smithi031_454752_object377:head# 2024-04-25T09:18:18.283 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.316 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:190ad911:::benchmark_data_smithi031_454752_object158:head# 2024-04-25T09:18:18.316 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.350 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:192b4b6c:::benchmark_data_smithi031_454752_object249:head# 2024-04-25T09:18:18.350 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.383 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1a2dbf6a:::benchmark_data_smithi031_454752_object284:head# 2024-04-25T09:18:18.383 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.417 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1bace3cc:::benchmark_data_smithi031_454752_object136:head# 2024-04-25T09:18:18.417 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.450 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1f6172c3:::benchmark_data_smithi031_454752_object315:head# 2024-04-25T09:18:18.450 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.483 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20988891:::benchmark_data_smithi031_454752_object0:head# 2024-04-25T09:18:18.483 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.517 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20c89311:::benchmark_data_smithi031_454752_object170:head# 2024-04-25T09:18:18.517 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.542 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20f96ee6:::benchmark_data_smithi031_454752_object30:head# 2024-04-25T09:18:18.542 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.568 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:210b81be:::benchmark_data_smithi031_454752_object396:head# 2024-04-25T09:18:18.568 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.592 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:222bd3b2:::benchmark_data_smithi031_454752_object244:head# 2024-04-25T09:18:18.592 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.617 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:23ea9d14:::benchmark_data_smithi031_454752_object134:head# 2024-04-25T09:18:18.617 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.642 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2564bee6:::benchmark_data_smithi031_454752_object68:head# 2024-04-25T09:18:18.642 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.667 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:26e5d41e:::benchmark_data_smithi031_454752_object261:head# 2024-04-25T09:18:18.668 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.693 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2765dc2c:::benchmark_data_smithi031_454752_object147:head# 2024-04-25T09:18:18.693 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.718 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:286a6080:::benchmark_data_smithi031_454752_object64:head# 2024-04-25T09:18:18.718 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.743 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2899a918:::benchmark_data_smithi031_454752_object267:head# 2024-04-25T09:18:18.743 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.770 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:299c9ca7:::benchmark_data_smithi031_454752_object21:head# 2024-04-25T09:18:18.770 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.793 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:29db69da:::benchmark_data_smithi031_454752_object236:head# 2024-04-25T09:18:18.793 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.818 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2aa178ac:::benchmark_data_smithi031_454752_object59:head# 2024-04-25T09:18:18.819 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.910 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ac07948:::benchmark_data_smithi031_454752_object7:head# 2024-04-25T09:18:18.911 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.952 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2af7345c:::benchmark_data_smithi031_454752_object202:head# 2024-04-25T09:18:18.952 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:18.979 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2c7cae34:::benchmark_data_smithi031_454752_object300:head# 2024-04-25T09:18:18.980 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.002 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2d714369:::benchmark_data_smithi031_454752_object292:head# 2024-04-25T09:18:19.002 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.027 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2da27532:::benchmark_data_smithi031_454752_object1:head# 2024-04-25T09:18:19.028 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.069 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2dee9738:::benchmark_data_smithi031_454752_object400:head# 2024-04-25T09:18:19.070 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.148 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e61c031:::benchmark_data_smithi031_454752_object174:head# 2024-04-25T09:18:19.148 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.212 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ee4a5f6:::benchmark_data_smithi031_454752_object94:head# 2024-04-25T09:18:19.212 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.237 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2f1e460a:::benchmark_data_smithi031_454752_object106:head# 2024-04-25T09:18:19.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.262 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2fc8a11e:::benchmark_data_smithi031_454752_object260:head# 2024-04-25T09:18:19.262 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.287 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:30797560:::benchmark_data_smithi031_454752_object142:head# 2024-04-25T09:18:19.287 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.312 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3105618d:::benchmark_data_smithi031_454752_object46:head# 2024-04-25T09:18:19.312 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.337 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31a53b00:::benchmark_data_smithi031_454752_object14:head# 2024-04-25T09:18:19.337 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.362 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31b8e0ab:::benchmark_data_smithi031_454752_object326:head# 2024-04-25T09:18:19.363 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.388 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31c961a5:::benchmark_data_smithi031_454752_object11:head# 2024-04-25T09:18:19.388 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.413 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31f4feb6:::benchmark_data_smithi031_454752_object54:head# 2024-04-25T09:18:19.413 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:332a7b27:::benchmark_data_smithi031_454752_object257:head# 2024-04-25T09:18:19.438 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.463 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:33cf1783:::benchmark_data_smithi031_454752_object83:head# 2024-04-25T09:18:19.463 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.488 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:33e69519:::benchmark_data_smithi031_454752_object334:head# 2024-04-25T09:18:19.488 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.513 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:343b464c:::benchmark_data_smithi031_454752_object229:head# 2024-04-25T09:18:19.514 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.538 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:353bc8d6:::benchmark_data_smithi031_454752_object306:head# 2024-04-25T09:18:19.539 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.572 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:357adf2e:::benchmark_data_smithi031_454752_object313:head# 2024-04-25T09:18:19.572 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.606 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3626a3d2:::benchmark_data_smithi031_454752_object242:head# 2024-04-25T09:18:19.606 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.639 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:366a3868:::benchmark_data_smithi031_454752_object344:head# 2024-04-25T09:18:19.639 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:36e1e4b7:::benchmark_data_smithi031_454752_object330:head# 2024-04-25T09:18:19.673 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.706 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3747f9e2:::benchmark_data_smithi031_454752_object361:head# 2024-04-25T09:18:19.706 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.739 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:376c5fd1:::benchmark_data_smithi031_454752_object276:head# 2024-04-25T09:18:19.740 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.773 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:37ad6f44:::benchmark_data_smithi031_454752_object199:head# 2024-04-25T09:18:19.773 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.808 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3809a39d:::benchmark_data_smithi031_454752_object63:head# 2024-04-25T09:18:19.808 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.840 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:38962184:::benchmark_data_smithi031_454752_object345:head# 2024-04-25T09:18:19.840 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.873 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3947aa59:::benchmark_data_smithi031_454752_object29:head# 2024-04-25T09:18:19.873 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.906 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:39a62bf7:::benchmark_data_smithi031_454752_object346:head# 2024-04-25T09:18:19.907 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.941 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3a70fad2:::benchmark_data_smithi031_454752_object213:head# 2024-04-25T09:18:19.941 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:19.973 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3aaccaa2:::benchmark_data_smithi031_454752_object245:head# 2024-04-25T09:18:19.973 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.007 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b4d69f0:::benchmark_data_smithi031_454752_object133:head# 2024-04-25T09:18:20.007 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.040 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b5ed4ad:::benchmark_data_smithi031_454752_object153:head# 2024-04-25T09:18:20.040 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.074 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3bd7c293:::benchmark_data_smithi031_454752_object226:head# 2024-04-25T09:18:20.074 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.107 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3d9bbce6:::benchmark_data_smithi031_454752_object195:head# 2024-04-25T09:18:20.107 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.157 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3e1dceb1:::benchmark_data_smithi031_454752_object163:head# 2024-04-25T09:18:20.158 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.191 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f585de7:::benchmark_data_smithi031_454752_object370:head# 2024-04-25T09:18:20.191 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.224 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f8dddc0:::benchmark_data_smithi031_454752_object36:head# 2024-04-25T09:18:20.224 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.258 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:40e869df:::benchmark_data_smithi031_454752_object354:head# 2024-04-25T09:18:20.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.291 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:42f70b52:::benchmark_data_smithi031_454752_object391:head# 2024-04-25T09:18:20.291 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.325 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:44648b7f:::benchmark_data_smithi031_454752_object15:head# 2024-04-25T09:18:20.325 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.358 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:45c0069e:::benchmark_data_smithi031_454752_object93:head# 2024-04-25T09:18:20.358 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.392 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:46cce1fb:::benchmark_data_smithi031_454752_object259:head# 2024-04-25T09:18:20.392 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.426 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48372743:::benchmark_data_smithi031_454752_object92:head# 2024-04-25T09:18:20.426 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.458 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:487de23f:::benchmark_data_smithi031_454752_object278:head# 2024-04-25T09:18:20.458 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.492 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48933b4f:::benchmark_data_smithi031_454752_object219:head# 2024-04-25T09:18:20.492 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.525 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48b43814:::benchmark_data_smithi031_454752_object22:head# 2024-04-25T09:18:20.526 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.559 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:49675aeb:::benchmark_data_smithi031_454752_object99:head# 2024-04-25T09:18:20.559 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.592 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a2d5412:::benchmark_data_smithi031_454752_object324:head# 2024-04-25T09:18:20.592 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.625 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a6e0d41:::benchmark_data_smithi031_454752_object268:head# 2024-04-25T09:18:20.626 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.659 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a6e92f8:::benchmark_data_smithi031_454752_object198:head# 2024-04-25T09:18:20.659 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.701 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a95c670:::benchmark_data_smithi031_454752_object220:head# 2024-04-25T09:18:20.701 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.734 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a9a32d5:::benchmark_data_smithi031_454752_object56:head# 2024-04-25T09:18:20.735 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.768 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4aa2f1b1:::benchmark_data_smithi031_454752_object160:head# 2024-04-25T09:18:20.768 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.793 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4ac6a300:::benchmark_data_smithi031_454752_object209:head# 2024-04-25T09:18:20.793 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.818 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b065b88:::benchmark_data_smithi031_454752_object34:head# 2024-04-25T09:18:20.818 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.893 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b3ae5fa:::benchmark_data_smithi031_454752_object399:head# 2024-04-25T09:18:20.893 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.935 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b491b4d:::benchmark_data_smithi031_454752_object86:head# 2024-04-25T09:18:20.935 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.960 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4c281e67:::benchmark_data_smithi031_454752_object123:head# 2024-04-25T09:18:20.960 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:20.985 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd1274c:::benchmark_data_smithi031_454752_object77:head# 2024-04-25T09:18:20.985 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.010 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd91fcf:::benchmark_data_smithi031_454752_object227:head# 2024-04-25T09:18:21.011 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.119 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd9e311:::benchmark_data_smithi031_454752_object165:head# 2024-04-25T09:18:21.119 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.165 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4e704b66:::benchmark_data_smithi031_454752_object110:head# 2024-04-25T09:18:21.165 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.229 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4f141352:::benchmark_data_smithi031_454752_object390:head# 2024-04-25T09:18:21.229 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.253 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4f67372e:::benchmark_data_smithi031_454752_object310:head# 2024-04-25T09:18:21.254 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.278 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5055bf66:::benchmark_data_smithi031_454752_object111:head# 2024-04-25T09:18:21.279 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.304 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5079fdc4:::benchmark_data_smithi031_454752_object102:head# 2024-04-25T09:18:21.304 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.337 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51592368:::benchmark_data_smithi031_454752_object129:head# 2024-04-25T09:18:21.337 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.362 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51c35831:::benchmark_data_smithi031_454752_object277:head# 2024-04-25T09:18:21.362 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.387 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:525ef634:::benchmark_data_smithi031_454752_object303:head# 2024-04-25T09:18:21.387 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.413 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:52f1a683:::benchmark_data_smithi031_454752_object232:head# 2024-04-25T09:18:21.413 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.438 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:549943cc:::benchmark_data_smithi031_454752_object137:head# 2024-04-25T09:18:21.438 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.463 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:54a7cad2:::benchmark_data_smithi031_454752_object243:head# 2024-04-25T09:18:21.463 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.488 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56c9c8a3:::benchmark_data_smithi031_454752_object386:head# 2024-04-25T09:18:21.488 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.530 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56d22a80:::benchmark_data_smithi031_454752_object61:head# 2024-04-25T09:18:21.530 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.555 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:576c3dbd:::benchmark_data_smithi031_454752_object320:head# 2024-04-25T09:18:21.556 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.580 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:580c5a46:::benchmark_data_smithi031_454752_object62:head# 2024-04-25T09:18:21.580 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.605 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:58815a7c:::benchmark_data_smithi031_454752_object392:head# 2024-04-25T09:18:21.605 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.630 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:58e6f983:::benchmark_data_smithi031_454752_object233:head# 2024-04-25T09:18:21.630 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.658 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:59c109e7:::benchmark_data_smithi031_454752_object372:head# 2024-04-25T09:18:21.658 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.681 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a7c4e16:::benchmark_data_smithi031_454752_object48:head# 2024-04-25T09:18:21.681 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.706 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a943612:::benchmark_data_smithi031_454752_object40:head# 2024-04-25T09:18:21.706 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.731 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5aed2644:::benchmark_data_smithi031_454752_object395:head# 2024-04-25T09:18:21.731 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.756 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5c81318e:::benchmark_data_smithi031_454752_object18:head# 2024-04-25T09:18:21.756 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.781 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5d21802c:::benchmark_data_smithi031_454752_object201:head# 2024-04-25T09:18:21.781 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.806 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5d50b842:::benchmark_data_smithi031_454752_object282:head# 2024-04-25T09:18:21.806 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.831 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5de1e511:::benchmark_data_smithi031_454752_object173:head# 2024-04-25T09:18:21.832 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.856 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e1e382c:::benchmark_data_smithi031_454752_object189:head# 2024-04-25T09:18:21.857 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.882 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e3603a8:::benchmark_data_smithi031_454752_object356:head# 2024-04-25T09:18:21.882 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.915 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e9a3526:::benchmark_data_smithi031_454752_object274:head# 2024-04-25T09:18:21.915 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.948 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:60dee496:::benchmark_data_smithi031_454752_object287:head# 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:21.982 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:610072df:::benchmark_data_smithi031_454752_object381:head# 2024-04-25T09:18:21.982 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.015 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:61757bc4:::benchmark_data_smithi031_454752_object100:head# 2024-04-25T09:18:22.015 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.092 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:61eb06a1:::benchmark_data_smithi031_454752_object42:head# 2024-04-25T09:18:22.092 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.124 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:622a095a:::benchmark_data_smithi031_454752_object172:head# 2024-04-25T09:18:22.124 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.157 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:629696e0:::benchmark_data_smithi031_454752_object343:head# 2024-04-25T09:18:22.158 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.199 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:62dff125:::benchmark_data_smithi031_454752_object117:head# 2024-04-25T09:18:22.199 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.241 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:637e2a8b:::benchmark_data_smithi031_454752_object124:head# 2024-04-25T09:18:22.241 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.275 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:63a84c81:::benchmark_data_smithi031_454752_object82:head# 2024-04-25T09:18:22.275 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.325 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:64a1f0ad:::benchmark_data_smithi031_454752_object367:head# 2024-04-25T09:18:22.325 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.358 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:65a082ea:::benchmark_data_smithi031_454752_object316:head# 2024-04-25T09:18:22.358 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.391 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:66d0a0c5:::benchmark_data_smithi031_454752_object4:head# 2024-04-25T09:18:22.392 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.425 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6757b3f7:::benchmark_data_smithi031_454752_object347:head# 2024-04-25T09:18:22.425 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.458 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68851dbf:::benchmark_data_smithi031_454752_object279:head# 2024-04-25T09:18:22.458 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.492 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68869d9f:::benchmark_data_smithi031_454752_object8:head# 2024-04-25T09:18:22.492 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6922a669:::benchmark_data_smithi031_454752_object291:head# 2024-04-25T09:18:22.525 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:695ddfd2:::benchmark_data_smithi031_454752_object285:head# 2024-04-25T09:18:22.559 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.592 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:69f748df:::benchmark_data_smithi031_454752_object380:head# 2024-04-25T09:18:22.593 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.625 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6a4d4ed2:::benchmark_data_smithi031_454752_object212:head# 2024-04-25T09:18:22.626 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.659 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d7df3bd:::benchmark_data_smithi031_454752_object323:head# 2024-04-25T09:18:22.659 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.693 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d9bcab7:::benchmark_data_smithi031_454752_object299:head# 2024-04-25T09:18:22.693 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.726 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6e1dd161:::benchmark_data_smithi031_454752_object176:head# 2024-04-25T09:18:22.726 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.760 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ea26cda:::benchmark_data_smithi031_454752_object237:head# 2024-04-25T09:18:22.760 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.801 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:715f84d9:::benchmark_data_smithi031_454752_object31:head# 2024-04-25T09:18:22.801 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.848 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:718b1888:::benchmark_data_smithi031_454752_object3:head# 2024-04-25T09:18:22.848 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.910 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7201ba57:::benchmark_data_smithi031_454752_object332:head# 2024-04-25T09:18:22.910 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.943 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72916a4f:::benchmark_data_smithi031_454752_object43:head# 2024-04-25T09:18:22.943 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:22.977 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72e5e7bc:::benchmark_data_smithi031_454752_object350:head# 2024-04-25T09:18:22.977 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.010 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72f2ccd2:::benchmark_data_smithi031_454752_object211:head# 2024-04-25T09:18:23.011 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.086 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:737dcd92:::benchmark_data_smithi031_454752_object335:head# 2024-04-25T09:18:23.086 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.144 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:744eb9c4:::benchmark_data_smithi031_454752_object101:head# 2024-04-25T09:18:23.145 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.186 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75451e3a:::benchmark_data_smithi031_454752_object215:head# 2024-04-25T09:18:23.187 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.212 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75b87d24:::benchmark_data_smithi031_454752_object156:head# 2024-04-25T09:18:23.212 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.237 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76a44be7:::benchmark_data_smithi031_454752_object371:head# 2024-04-25T09:18:23.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.262 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76a5bace:::benchmark_data_smithi031_454752_object253:head# 2024-04-25T09:18:23.262 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.329 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:772eee67:::benchmark_data_smithi031_454752_object121:head# 2024-04-25T09:18:23.329 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.354 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:798c5654:::benchmark_data_smithi031_454752_object204:head# 2024-04-25T09:18:23.354 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.379 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7995db24:::benchmark_data_smithi031_454752_object157:head# 2024-04-25T09:18:23.379 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.404 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79abc611:::benchmark_data_smithi031_454752_object164:head# 2024-04-25T09:18:23.404 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.429 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79d1490c:::benchmark_data_smithi031_454752_object138:head# 2024-04-25T09:18:23.429 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.454 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7b6edc17:::benchmark_data_smithi031_454752_object214:head# 2024-04-25T09:18:23.454 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.479 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7b93ca23:::benchmark_data_smithi031_454752_object55:head# 2024-04-25T09:18:23.479 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.504 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bd5320a:::benchmark_data_smithi031_454752_object328:head# 2024-04-25T09:18:23.504 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.529 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bde86cb:::benchmark_data_smithi031_454752_object182:head# 2024-04-25T09:18:23.530 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.554 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bfed9be:::benchmark_data_smithi031_454752_object397:head# 2024-04-25T09:18:23.555 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.588 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7c0b3015:::benchmark_data_smithi031_454752_object108:head# 2024-04-25T09:18:23.588 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.613 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7cb420f8:::benchmark_data_smithi031_454752_object89:head# 2024-04-25T09:18:23.613 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.638 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7d588f87:::benchmark_data_smithi031_454752_object376:head# 2024-04-25T09:18:23.639 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.663 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dbcb22e:::benchmark_data_smithi031_454752_object17:head# 2024-04-25T09:18:23.664 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.688 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dc6b3de:::benchmark_data_smithi031_454752_object187:head# 2024-04-25T09:18:23.689 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.714 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dec0b89:::benchmark_data_smithi031_454752_object149:head# 2024-04-25T09:18:23.714 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.739 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dfe5ddd:::benchmark_data_smithi031_454752_object79:head# 2024-04-25T09:18:23.739 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.764 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7ea5a1e2:::benchmark_data_smithi031_454752_object360:head# 2024-04-25T09:18:23.764 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.789 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7eeb5bde:::benchmark_data_smithi031_454752_object186:head# 2024-04-25T09:18:23.789 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.814 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7fb12f34:::benchmark_data_smithi031_454752_object69:head# 2024-04-25T09:18:23.814 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:23.915 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 10 info 1.0( v 10'213 (0'0,10'213] local-lis/les=7/8 n=213 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:18:23.981 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T09:18:24.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:18:25.662 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:18:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:18:25.662 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:18:26.027 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:18:27.500 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:18:27.500 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:18:27.500 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:18:27.797 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T09:18:28.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-04-25T09:18:29.577 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:18:29.577 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T09:18:29.578 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0005830b:::benchmark_data_smithi031_454752_object169:head# 2024-04-25T09:18:29.578 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:29.719 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0007ff59:::benchmark_data_smithi031_454752_object384:head# 2024-04-25T09:18:29.719 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:29.811 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00530cad:::benchmark_data_smithi031_454752_object366:head# 2024-04-25T09:18:29.811 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:29.878 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:007182ad:::benchmark_data_smithi031_454752_object150:head# 2024-04-25T09:18:29.878 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:29.936 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00808219:::benchmark_data_smithi031_454752_object10:head# 2024-04-25T09:18:29.936 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:29.995 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00ccfbba:::benchmark_data_smithi031_454752_object355:head# 2024-04-25T09:18:29.995 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.053 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00d7eda8:::benchmark_data_smithi031_454752_object357:head# 2024-04-25T09:18:30.053 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.120 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:019d4e67:::benchmark_data_smithi031_460268_object76:head# 2024-04-25T09:18:30.120 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.153 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:01faf234:::benchmark_data_smithi031_454752_object272:head# 2024-04-25T09:18:30.154 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.187 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:021d5fe5:::benchmark_data_smithi031_454752_object109:head# 2024-04-25T09:18:30.187 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.220 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:023d725e:::benchmark_data_smithi031_460268_object88:head# 2024-04-25T09:18:30.221 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.263 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:02a87b95:::benchmark_data_smithi031_454752_object26:head# 2024-04-25T09:18:30.263 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.296 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:02bf8c5e:::benchmark_data_smithi031_460268_object64:head# 2024-04-25T09:18:30.296 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.329 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03119e67:::benchmark_data_smithi031_454752_object120:head# 2024-04-25T09:18:30.330 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.363 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:032c7ba3:::benchmark_data_smithi031_460268_object73:head# 2024-04-25T09:18:30.363 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.396 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03a19a01:::benchmark_data_smithi031_454752_object375:head# 2024-04-25T09:18:30.397 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.430 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:03d8d0d6:::benchmark_data_smithi031_454752_object307:head# 2024-04-25T09:18:30.430 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.463 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0407316c:::benchmark_data_smithi031_454752_object248:head# 2024-04-25T09:18:30.463 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.496 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:04e37833:::benchmark_data_smithi031_460268_object95:head# 2024-04-25T09:18:30.496 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.530 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:05c088cd:::benchmark_data_smithi031_460268_object132:head# 2024-04-25T09:18:30.530 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.563 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:069ac774:::benchmark_data_smithi031_454752_object270:head# 2024-04-25T09:18:30.564 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.597 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:076ac3b3:::benchmark_data_smithi031_454752_object168:head# 2024-04-25T09:18:30.597 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.630 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:078ca8f4:::benchmark_data_smithi031_454752_object271:head# 2024-04-25T09:18:30.631 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.664 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:089737ee:::benchmark_data_smithi031_454752_object262:head# 2024-04-25T09:18:30.664 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.697 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:08fea58a:::benchmark_data_smithi031_460268_object40:head# 2024-04-25T09:18:30.697 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.731 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0954f670:::benchmark_data_smithi031_454752_object223:head# 2024-04-25T09:18:30.891 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.892 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:09d319a2:::benchmark_data_smithi031_454752_object283:head# 2024-04-25T09:18:30.892 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.893 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a33ee0a:::benchmark_data_smithi031_454752_object107:head# 2024-04-25T09:18:30.893 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.893 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0a588e5f:::benchmark_data_smithi031_454752_object305:head# 2024-04-25T09:18:30.893 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.906 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ae0c865:::benchmark_data_smithi031_454752_object41:head# 2024-04-25T09:18:30.907 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.957 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0c38e0a3:::benchmark_data_smithi031_454752_object387:head# 2024-04-25T09:18:30.957 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0c4decad:::benchmark_data_smithi031_454752_object151:head# 2024-04-25T09:18:30.990 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.023 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0cc7bcb7:::benchmark_data_smithi031_454752_object333:head# 2024-04-25T09:18:31.024 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.065 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ce7b7bc:::benchmark_data_smithi031_454752_object353:head# 2024-04-25T09:18:31.066 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.144 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0d6e3436:::benchmark_data_smithi031_454752_object205:head# 2024-04-25T09:18:31.144 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.208 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0dd158a0:::benchmark_data_smithi031_454752_object76:head# 2024-04-25T09:18:31.208 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.241 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0df905cb:::benchmark_data_smithi031_454752_object181:head# 2024-04-25T09:18:31.241 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.275 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0e56a830:::benchmark_data_smithi031_460268_object58:head# 2024-04-25T09:18:31.275 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.308 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ec7061f:::benchmark_data_smithi031_460268_object9:head# 2024-04-25T09:18:31.308 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.342 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0f047668:::benchmark_data_smithi031_454752_object12:head# 2024-04-25T09:18:31.342 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.375 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0f62c75d:::benchmark_data_smithi031_460268_object111:head# 2024-04-25T09:18:31.375 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.408 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:100b6e67:::benchmark_data_smithi031_454752_object122:head# 2024-04-25T09:18:31.409 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.442 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:105e9b29:::benchmark_data_smithi031_454752_object295:head# 2024-04-25T09:18:31.442 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.475 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:10ca342c:::benchmark_data_smithi031_454752_object203:head# 2024-04-25T09:18:31.475 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.509 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:10d8c220:::benchmark_data_smithi031_460268_object108:head# 2024-04-25T09:18:31.509 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.542 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:117584c9:::benchmark_data_smithi031_460268_object80:head# 2024-04-25T09:18:31.542 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.576 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:11985b73:::benchmark_data_smithi031_454752_object294:head# 2024-04-25T09:18:31.576 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.609 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1215ebb0:::benchmark_data_smithi031_454752_object96:head# 2024-04-25T09:18:31.609 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.643 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:123f25d6:::benchmark_data_smithi031_454752_object339:head# 2024-04-25T09:18:31.643 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.676 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:12648b62:::benchmark_data_smithi031_454752_object235:head# 2024-04-25T09:18:31.676 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.710 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:13b45992:::benchmark_data_smithi031_460268_object5:head# 2024-04-25T09:18:31.710 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.743 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:13b5933b:::benchmark_data_smithi031_454752_object258:head# 2024-04-25T09:18:31.743 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.793 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:14a4f03a:::benchmark_data_smithi031_454752_object90:head# 2024-04-25T09:18:31.793 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.826 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:14f470c1:::benchmark_data_smithi031_460268_object61:head# 2024-04-25T09:18:31.827 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.860 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:152756a2:::benchmark_data_smithi031_454752_object78:head# 2024-04-25T09:18:31.860 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.893 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:16b02634:::benchmark_data_smithi031_460268_object169:head# 2024-04-25T09:18:31.894 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.927 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:174b1785:::benchmark_data_smithi031_460268_object49:head# 2024-04-25T09:18:31.927 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.960 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17685eef:::benchmark_data_smithi031_454752_object35:head# 2024-04-25T09:18:31.961 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:31.994 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:17d78309:::benchmark_data_smithi031_454752_object197:head# 2024-04-25T09:18:31.994 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.027 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:186323b7:::benchmark_data_smithi031_454752_object264:head# 2024-04-25T09:18:32.027 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.061 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18a96ee0:::benchmark_data_smithi031_454752_object340:head# 2024-04-25T09:18:32.061 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.094 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18abc847:::benchmark_data_smithi031_454752_object377:head# 2024-04-25T09:18:32.094 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.128 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:190ad911:::benchmark_data_smithi031_454752_object158:head# 2024-04-25T09:18:32.128 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.169 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:192b4b6c:::benchmark_data_smithi031_454752_object249:head# 2024-04-25T09:18:32.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.211 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:19e0e3b2:::benchmark_data_smithi031_460268_object175:head# 2024-04-25T09:18:32.211 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.278 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1a2dbf6a:::benchmark_data_smithi031_454752_object284:head# 2024-04-25T09:18:32.278 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.328 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1a523430:::benchmark_data_smithi031_460268_object184:head# 2024-04-25T09:18:32.329 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.370 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1ba28391:::benchmark_data_smithi031_460268_object129:head# 2024-04-25T09:18:32.370 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.412 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1bace3cc:::benchmark_data_smithi031_454752_object136:head# 2024-04-25T09:18:32.412 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.454 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1c390819:::benchmark_data_smithi031_460268_object149:head# 2024-04-25T09:18:32.454 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.495 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1da2bdd0:::benchmark_data_smithi031_460268_object168:head# 2024-04-25T09:18:32.495 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.538 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1e87b9f0:::benchmark_data_smithi031_460268_object130:head# 2024-04-25T09:18:32.538 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.571 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1f6172c3:::benchmark_data_smithi031_454752_object315:head# 2024-04-25T09:18:32.571 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.604 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1f9ba3a6:::benchmark_data_smithi031_460268_object191:head# 2024-04-25T09:18:32.604 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.638 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20988891:::benchmark_data_smithi031_454752_object0:head# 2024-04-25T09:18:32.638 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.671 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20c89311:::benchmark_data_smithi031_454752_object170:head# 2024-04-25T09:18:32.671 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:20f96ee6:::benchmark_data_smithi031_454752_object30:head# 2024-04-25T09:18:32.705 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:210b81be:::benchmark_data_smithi031_454752_object396:head# 2024-04-25T09:18:32.739 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.771 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:222bd3b2:::benchmark_data_smithi031_454752_object244:head# 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.805 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:23439e48:::benchmark_data_smithi031_460268_object90:head# 2024-04-25T09:18:32.805 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.849 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:23ea9d14:::benchmark_data_smithi031_454752_object134:head# 2024-04-25T09:18:32.849 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.939 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2564bee6:::benchmark_data_smithi031_454752_object68:head# 2024-04-25T09:18:32.939 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:32.972 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2587b006:::benchmark_data_smithi031_460268_object114:head# 2024-04-25T09:18:32.972 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.006 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:259c03b1:::benchmark_data_smithi031_460268_object44:head# 2024-04-25T09:18:33.006 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.049 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:26e5d41e:::benchmark_data_smithi031_454752_object261:head# 2024-04-25T09:18:33.049 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.144 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2765dc2c:::benchmark_data_smithi031_454752_object147:head# 2024-04-25T09:18:33.144 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.207 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2805b1eb:::benchmark_data_smithi031_460268_object10:head# 2024-04-25T09:18:33.207 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.240 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2826ee59:::benchmark_data_smithi031_460268_object1:head# 2024-04-25T09:18:33.240 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.274 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:286a6080:::benchmark_data_smithi031_454752_object64:head# 2024-04-25T09:18:33.274 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.307 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2899a918:::benchmark_data_smithi031_454752_object267:head# 2024-04-25T09:18:33.307 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.340 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:293ed6b9:::benchmark_data_smithi031_460268_object55:head# 2024-04-25T09:18:33.341 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.374 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2972c7c4:::benchmark_data_smithi031_460268_object141:head# 2024-04-25T09:18:33.375 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.407 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:299c9ca7:::benchmark_data_smithi031_454752_object21:head# 2024-04-25T09:18:33.407 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.441 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:29db69da:::benchmark_data_smithi031_454752_object236:head# 2024-04-25T09:18:33.441 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.474 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:29dfd43b:::benchmark_data_smithi031_460268_object144:head# 2024-04-25T09:18:33.474 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.508 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2aa178ac:::benchmark_data_smithi031_454752_object59:head# 2024-04-25T09:18:33.508 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.541 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ac07948:::benchmark_data_smithi031_454752_object7:head# 2024-04-25T09:18:33.541 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.575 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2af7345c:::benchmark_data_smithi031_454752_object202:head# 2024-04-25T09:18:33.575 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.641 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2bd9629a:::benchmark_data_smithi031_460268_object68:head# 2024-04-25T09:18:33.641 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.675 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2c7cae34:::benchmark_data_smithi031_454752_object300:head# 2024-04-25T09:18:33.675 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.715 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2d714369:::benchmark_data_smithi031_454752_object292:head# 2024-04-25T09:18:33.715 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.804 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2da27532:::benchmark_data_smithi031_454752_object1:head# 2024-04-25T09:18:33.804 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.876 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2dde69f0:::benchmark_data_smithi031_460268_object103:head# 2024-04-25T09:18:33.876 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:33.943 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2dee9738:::benchmark_data_smithi031_454752_object400:head# 2024-04-25T09:18:33.943 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.027 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e578dfe:::benchmark_data_smithi031_460268_object86:head# 2024-04-25T09:18:34.027 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.077 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e61c031:::benchmark_data_smithi031_454752_object174:head# 2024-04-25T09:18:34.077 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.127 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ee4a5f6:::benchmark_data_smithi031_454752_object94:head# 2024-04-25T09:18:34.127 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.162 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2f1e460a:::benchmark_data_smithi031_454752_object106:head# 2024-04-25T09:18:34.163 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.194 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2f48312c:::benchmark_data_smithi031_460268_object127:head# 2024-04-25T09:18:34.194 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.228 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2fb10bd8:::benchmark_data_smithi031_460268_object159:head# 2024-04-25T09:18:34.228 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.261 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2fc8a11e:::benchmark_data_smithi031_454752_object260:head# 2024-04-25T09:18:34.261 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.294 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2fcd61f1:::benchmark_data_smithi031_460268_object140:head# 2024-04-25T09:18:34.295 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.328 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:30797560:::benchmark_data_smithi031_454752_object142:head# 2024-04-25T09:18:34.328 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.361 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3105618d:::benchmark_data_smithi031_454752_object46:head# 2024-04-25T09:18:34.362 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.395 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3136920f:::benchmark_data_smithi031_460268_object131:head# 2024-04-25T09:18:34.395 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.428 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31a53b00:::benchmark_data_smithi031_454752_object14:head# 2024-04-25T09:18:34.428 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.462 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31b8e0ab:::benchmark_data_smithi031_454752_object326:head# 2024-04-25T09:18:34.462 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.495 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31c961a5:::benchmark_data_smithi031_454752_object11:head# 2024-04-25T09:18:34.495 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.529 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31f4feb6:::benchmark_data_smithi031_454752_object54:head# 2024-04-25T09:18:34.529 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.571 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:32147e98:::benchmark_data_smithi031_460268_object151:head# 2024-04-25T09:18:34.571 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.612 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3295aa64:::benchmark_data_smithi031_460268_object46:head# 2024-04-25T09:18:34.612 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.654 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:332a7b27:::benchmark_data_smithi031_454752_object257:head# 2024-04-25T09:18:34.654 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.696 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:33cf1783:::benchmark_data_smithi031_454752_object83:head# 2024-04-25T09:18:34.696 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:33e69519:::benchmark_data_smithi031_454752_object334:head# 2024-04-25T09:18:34.738 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.779 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:343b464c:::benchmark_data_smithi031_454752_object229:head# 2024-04-25T09:18:34.780 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.821 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:34fc845a:::benchmark_data_smithi031_460268_object146:head# 2024-04-25T09:18:34.821 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.880 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:353bc8d6:::benchmark_data_smithi031_454752_object306:head# 2024-04-25T09:18:34.880 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.930 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:355e1d34:::benchmark_data_smithi031_460268_object75:head# 2024-04-25T09:18:34.930 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:34.964 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:357adf2e:::benchmark_data_smithi031_454752_object313:head# 2024-04-25T09:18:34.964 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.056 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3626a3d2:::benchmark_data_smithi031_454752_object242:head# 2024-04-25T09:18:35.056 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.126 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:366a3868:::benchmark_data_smithi031_454752_object344:head# 2024-04-25T09:18:35.127 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.199 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:36e1e4b7:::benchmark_data_smithi031_454752_object330:head# 2024-04-25T09:18:35.199 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.232 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3747f9e2:::benchmark_data_smithi031_454752_object361:head# 2024-04-25T09:18:35.232 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.265 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:376c5fd1:::benchmark_data_smithi031_454752_object276:head# 2024-04-25T09:18:35.265 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.298 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:37ad6f44:::benchmark_data_smithi031_454752_object199:head# 2024-04-25T09:18:35.299 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.332 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3809a39d:::benchmark_data_smithi031_454752_object63:head# 2024-04-25T09:18:35.332 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.365 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3861888a:::benchmark_data_smithi031_460268_object109:head# 2024-04-25T09:18:35.365 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.407 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:38962184:::benchmark_data_smithi031_454752_object345:head# 2024-04-25T09:18:35.407 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.440 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3947aa59:::benchmark_data_smithi031_454752_object29:head# 2024-04-25T09:18:35.441 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.474 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:39a62bf7:::benchmark_data_smithi031_454752_object346:head# 2024-04-25T09:18:35.474 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.524 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3a70fad2:::benchmark_data_smithi031_454752_object213:head# 2024-04-25T09:18:35.524 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.558 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3aaccaa2:::benchmark_data_smithi031_454752_object245:head# 2024-04-25T09:18:35.558 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.595 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3acaee8b:::benchmark_data_smithi031_460268_object48:head# 2024-04-25T09:18:35.595 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.624 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b4d69f0:::benchmark_data_smithi031_454752_object133:head# 2024-04-25T09:18:35.625 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.658 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b5ed4ad:::benchmark_data_smithi031_454752_object153:head# 2024-04-25T09:18:35.658 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.691 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3bd7c293:::benchmark_data_smithi031_454752_object226:head# 2024-04-25T09:18:35.691 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.725 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3d9bbce6:::benchmark_data_smithi031_454752_object195:head# 2024-04-25T09:18:35.725 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.758 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3e1dceb1:::benchmark_data_smithi031_454752_object163:head# 2024-04-25T09:18:35.759 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.792 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f585de7:::benchmark_data_smithi031_454752_object370:head# 2024-04-25T09:18:35.792 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.826 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f8dddc0:::benchmark_data_smithi031_454752_object36:head# 2024-04-25T09:18:35.826 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.859 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:40e869df:::benchmark_data_smithi031_454752_object354:head# 2024-04-25T09:18:35.859 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.892 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:42077408:::benchmark_data_smithi031_460268_object138:head# 2024-04-25T09:18:35.892 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.925 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4240854c:::benchmark_data_smithi031_460268_object99:head# 2024-04-25T09:18:35.926 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.959 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:429d8266:::benchmark_data_smithi031_460268_object157:head# 2024-04-25T09:18:35.959 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:35.993 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:42f70b52:::benchmark_data_smithi031_454752_object391:head# 2024-04-25T09:18:35.993 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.034 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:42fae8c1:::benchmark_data_smithi031_460268_object158:head# 2024-04-25T09:18:36.035 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.067 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:44648b7f:::benchmark_data_smithi031_454752_object15:head# 2024-04-25T09:18:36.068 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.101 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:44e34ed7:::benchmark_data_smithi031_460268_object47:head# 2024-04-25T09:18:36.101 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.143 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:45c0069e:::benchmark_data_smithi031_454752_object93:head# 2024-04-25T09:18:36.143 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.176 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:45d9ab4f:::benchmark_data_smithi031_460268_object79:head# 2024-04-25T09:18:36.177 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.211 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:46cce1fb:::benchmark_data_smithi031_454752_object259:head# 2024-04-25T09:18:36.211 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.243 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48372743:::benchmark_data_smithi031_454752_object92:head# 2024-04-25T09:18:36.243 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.277 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:487de23f:::benchmark_data_smithi031_454752_object278:head# 2024-04-25T09:18:36.277 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.310 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48933b4f:::benchmark_data_smithi031_454752_object219:head# 2024-04-25T09:18:36.310 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.352 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:48b43814:::benchmark_data_smithi031_454752_object22:head# 2024-04-25T09:18:36.352 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.394 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:49487ec1:::benchmark_data_smithi031_460268_object62:head# 2024-04-25T09:18:36.394 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.427 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:49675aeb:::benchmark_data_smithi031_454752_object99:head# 2024-04-25T09:18:36.427 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.460 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a2d5412:::benchmark_data_smithi031_454752_object324:head# 2024-04-25T09:18:36.461 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.494 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a6e0d41:::benchmark_data_smithi031_454752_object268:head# 2024-04-25T09:18:36.494 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.527 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a6e92f8:::benchmark_data_smithi031_454752_object198:head# 2024-04-25T09:18:36.528 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.569 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a95c670:::benchmark_data_smithi031_454752_object220:head# 2024-04-25T09:18:36.569 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.603 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4a9a32d5:::benchmark_data_smithi031_454752_object56:head# 2024-04-25T09:18:36.603 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.636 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4aa2f1b1:::benchmark_data_smithi031_454752_object160:head# 2024-04-25T09:18:36.636 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.669 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4ac6a300:::benchmark_data_smithi031_454752_object209:head# 2024-04-25T09:18:36.669 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.703 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4af7d290:::benchmark_data_smithi031_460268_object97:head# 2024-04-25T09:18:36.703 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.736 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b065b88:::benchmark_data_smithi031_454752_object34:head# 2024-04-25T09:18:36.736 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.770 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b199cf4:::benchmark_data_smithi031_460268_object102:head# 2024-04-25T09:18:36.770 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.803 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b3ae5fa:::benchmark_data_smithi031_454752_object399:head# 2024-04-25T09:18:36.804 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.845 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b491b4d:::benchmark_data_smithi031_454752_object86:head# 2024-04-25T09:18:36.845 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.929 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b70dd42:::benchmark_data_smithi031_460268_object133:head# 2024-04-25T09:18:36.929 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.962 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4c281e67:::benchmark_data_smithi031_454752_object123:head# 2024-04-25T09:18:36.963 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:36.996 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cc52962:::benchmark_data_smithi031_460268_object122:head# 2024-04-25T09:18:36.996 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.038 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd1274c:::benchmark_data_smithi031_454752_object77:head# 2024-04-25T09:18:37.038 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.120 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd91fcf:::benchmark_data_smithi031_454752_object227:head# 2024-04-25T09:18:37.120 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4cd9e311:::benchmark_data_smithi031_454752_object165:head# 2024-04-25T09:18:37.189 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.230 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4e704b66:::benchmark_data_smithi031_454752_object110:head# 2024-04-25T09:18:37.231 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.272 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4f141352:::benchmark_data_smithi031_454752_object390:head# 2024-04-25T09:18:37.272 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4f67372e:::benchmark_data_smithi031_454752_object310:head# 2024-04-25T09:18:37.314 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.347 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5055bf66:::benchmark_data_smithi031_454752_object111:head# 2024-04-25T09:18:37.348 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.381 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:505bbf46:::benchmark_data_smithi031_460268_object180:head# 2024-04-25T09:18:37.381 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.414 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5079fdc4:::benchmark_data_smithi031_454752_object102:head# 2024-04-25T09:18:37.414 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.473 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51592368:::benchmark_data_smithi031_454752_object129:head# 2024-04-25T09:18:37.473 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51c35831:::benchmark_data_smithi031_454752_object277:head# 2024-04-25T09:18:37.506 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.573 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:525ef634:::benchmark_data_smithi031_454752_object303:head# 2024-04-25T09:18:37.573 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.607 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:526ebac4:::benchmark_data_smithi031_460268_object172:head# 2024-04-25T09:18:37.607 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.640 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:52f1a683:::benchmark_data_smithi031_454752_object232:head# 2024-04-25T09:18:37.641 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.673 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:539a5e15:::benchmark_data_smithi031_460268_object96:head# 2024-04-25T09:18:37.674 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.707 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:53c14896:::benchmark_data_smithi031_460268_object29:head# 2024-04-25T09:18:37.707 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.740 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:549943cc:::benchmark_data_smithi031_454752_object137:head# 2024-04-25T09:18:37.740 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.774 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:54a7cad2:::benchmark_data_smithi031_454752_object243:head# 2024-04-25T09:18:37.774 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.807 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56294a42:::benchmark_data_smithi031_460268_object125:head# 2024-04-25T09:18:37.807 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.841 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:565ee0b3:::benchmark_data_smithi031_460268_object87:head# 2024-04-25T09:18:37.841 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.874 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56c9c8a3:::benchmark_data_smithi031_454752_object386:head# 2024-04-25T09:18:37.874 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.907 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56d22a80:::benchmark_data_smithi031_454752_object61:head# 2024-04-25T09:18:37.907 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.941 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57498d87:::benchmark_data_smithi031_460268_object15:head# 2024-04-25T09:18:37.941 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:37.974 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57503a76:::benchmark_data_smithi031_460268_object177:head# 2024-04-25T09:18:37.974 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.008 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:576c3dbd:::benchmark_data_smithi031_454752_object320:head# 2024-04-25T09:18:38.008 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.041 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57e97d52:::benchmark_data_smithi031_460268_object152:head# 2024-04-25T09:18:38.042 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.075 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57f53672:::benchmark_data_smithi031_460268_object137:head# 2024-04-25T09:18:38.075 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.108 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:580c5a46:::benchmark_data_smithi031_454752_object62:head# 2024-04-25T09:18:38.108 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:58815a7c:::benchmark_data_smithi031_454752_object392:head# 2024-04-25T09:18:38.142 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.175 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:58e6f983:::benchmark_data_smithi031_454752_object233:head# 2024-04-25T09:18:38.175 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.209 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:59c109e7:::benchmark_data_smithi031_454752_object372:head# 2024-04-25T09:18:38.209 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.242 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a0f0b43:::benchmark_data_smithi031_460268_object19:head# 2024-04-25T09:18:38.242 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.276 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a7c4e16:::benchmark_data_smithi031_454752_object48:head# 2024-04-25T09:18:38.276 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.309 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a943612:::benchmark_data_smithi031_454752_object40:head# 2024-04-25T09:18:38.309 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.342 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5aed2644:::benchmark_data_smithi031_454752_object395:head# 2024-04-25T09:18:38.342 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.376 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5b4543d1:::benchmark_data_smithi031_460268_object98:head# 2024-04-25T09:18:38.376 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.409 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5bae9bea:::benchmark_data_smithi031_460268_object165:head# 2024-04-25T09:18:38.409 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.443 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5c81318e:::benchmark_data_smithi031_454752_object18:head# 2024-04-25T09:18:38.443 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.476 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5c9e0861:::benchmark_data_smithi031_460268_object178:head# 2024-04-25T09:18:38.476 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.510 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5cde96ce:::benchmark_data_smithi031_460268_object164:head# 2024-04-25T09:18:38.510 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.543 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5d21802c:::benchmark_data_smithi031_454752_object201:head# 2024-04-25T09:18:38.543 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.577 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5d50b842:::benchmark_data_smithi031_454752_object282:head# 2024-04-25T09:18:38.577 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.610 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5de1e511:::benchmark_data_smithi031_454752_object173:head# 2024-04-25T09:18:38.610 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.643 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e1e382c:::benchmark_data_smithi031_454752_object189:head# 2024-04-25T09:18:38.643 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.677 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e3603a8:::benchmark_data_smithi031_454752_object356:head# 2024-04-25T09:18:38.677 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.710 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e9091db:::benchmark_data_smithi031_460268_object41:head# 2024-04-25T09:18:38.710 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.744 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e9a3526:::benchmark_data_smithi031_454752_object274:head# 2024-04-25T09:18:38.744 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.777 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5f2d69c6:::benchmark_data_smithi031_460268_object0:head# 2024-04-25T09:18:38.777 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.811 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5fb49c8b:::benchmark_data_smithi031_460268_object160:head# 2024-04-25T09:18:38.811 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.853 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:60047f02:::benchmark_data_smithi031_460268_object69:head# 2024-04-25T09:18:38.853 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.936 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:60dee496:::benchmark_data_smithi031_454752_object287:head# 2024-04-25T09:18:38.936 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:38.970 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:610072df:::benchmark_data_smithi031_454752_object381:head# 2024-04-25T09:18:38.970 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.003 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:61757bc4:::benchmark_data_smithi031_454752_object100:head# 2024-04-25T09:18:39.003 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.045 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:61eb06a1:::benchmark_data_smithi031_454752_object42:head# 2024-04-25T09:18:39.045 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.137 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:621c25df:::benchmark_data_smithi031_460268_object123:head# 2024-04-25T09:18:39.137 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.204 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:622a095a:::benchmark_data_smithi031_454752_object172:head# 2024-04-25T09:18:39.205 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.238 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:629696e0:::benchmark_data_smithi031_454752_object343:head# 2024-04-25T09:18:39.238 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.271 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:62b15a24:::benchmark_data_smithi031_460268_object107:head# 2024-04-25T09:18:39.271 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.305 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:62dff125:::benchmark_data_smithi031_454752_object117:head# 2024-04-25T09:18:39.305 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.338 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:637e2a8b:::benchmark_data_smithi031_454752_object124:head# 2024-04-25T09:18:39.338 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.372 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:63a84c81:::benchmark_data_smithi031_454752_object82:head# 2024-04-25T09:18:39.372 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.405 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:645c6c7a:::benchmark_data_smithi031_460268_object60:head# 2024-04-25T09:18:39.405 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.438 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:64a1f0ad:::benchmark_data_smithi031_454752_object367:head# 2024-04-25T09:18:39.439 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.472 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:650a3266:::benchmark_data_smithi031_460268_object77:head# 2024-04-25T09:18:39.472 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.505 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6554a2d5:::benchmark_data_smithi031_460268_object37:head# 2024-04-25T09:18:39.506 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.539 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:65a082ea:::benchmark_data_smithi031_454752_object316:head# 2024-04-25T09:18:39.539 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.572 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:66d0a0c5:::benchmark_data_smithi031_454752_object4:head# 2024-04-25T09:18:39.572 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.606 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:671a3d79:::benchmark_data_smithi031_460268_object70:head# 2024-04-25T09:18:39.606 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.639 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:675475fc:::benchmark_data_smithi031_460268_object143:head# 2024-04-25T09:18:39.639 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.681 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6757b3f7:::benchmark_data_smithi031_454752_object347:head# 2024-04-25T09:18:39.681 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.723 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:67d2de21:::benchmark_data_smithi031_460268_object11:head# 2024-04-25T09:18:39.723 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.765 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68851dbf:::benchmark_data_smithi031_454752_object279:head# 2024-04-25T09:18:39.765 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.806 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68869d9f:::benchmark_data_smithi031_454752_object8:head# 2024-04-25T09:18:39.806 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.848 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:68a2186c:::benchmark_data_smithi031_460268_object121:head# 2024-04-25T09:18:39.848 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.890 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6922a669:::benchmark_data_smithi031_454752_object291:head# 2024-04-25T09:18:39.890 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.949 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:695ddfd2:::benchmark_data_smithi031_454752_object285:head# 2024-04-25T09:18:39.949 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:39.982 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:69f748df:::benchmark_data_smithi031_454752_object380:head# 2024-04-25T09:18:39.982 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.015 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:69fb4598:::benchmark_data_smithi031_460268_object176:head# 2024-04-25T09:18:40.016 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.049 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6a4d4ed2:::benchmark_data_smithi031_454752_object212:head# 2024-04-25T09:18:40.049 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.082 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6b91f5bc:::benchmark_data_smithi031_460268_object155:head# 2024-04-25T09:18:40.082 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.116 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ca35829:::benchmark_data_smithi031_460268_object167:head# 2024-04-25T09:18:40.116 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.149 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ce64eae:::benchmark_data_smithi031_460268_object21:head# 2024-04-25T09:18:40.149 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.182 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d59ce53:::benchmark_data_smithi031_460268_object54:head# 2024-04-25T09:18:40.183 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d7df3bd:::benchmark_data_smithi031_454752_object323:head# 2024-04-25T09:18:40.216 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.249 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d9bcab7:::benchmark_data_smithi031_454752_object299:head# 2024-04-25T09:18:40.250 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.283 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6e1dd161:::benchmark_data_smithi031_454752_object176:head# 2024-04-25T09:18:40.283 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.316 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6e68daeb:::benchmark_data_smithi031_460268_object135:head# 2024-04-25T09:18:40.317 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.350 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ea26cda:::benchmark_data_smithi031_454752_object237:head# 2024-04-25T09:18:40.350 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.383 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6f7ed7e9:::benchmark_data_smithi031_460268_object26:head# 2024-04-25T09:18:40.383 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.417 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6f89a95c:::benchmark_data_smithi031_460268_object148:head# 2024-04-25T09:18:40.417 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.450 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:70e2520b:::benchmark_data_smithi031_460268_object170:head# 2024-04-25T09:18:40.450 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.484 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7146e359:::benchmark_data_smithi031_460268_object116:head# 2024-04-25T09:18:40.484 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.518 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:715f84d9:::benchmark_data_smithi031_454752_object31:head# 2024-04-25T09:18:40.519 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.550 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:718b1888:::benchmark_data_smithi031_454752_object3:head# 2024-04-25T09:18:40.551 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.584 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7201ba57:::benchmark_data_smithi031_454752_object332:head# 2024-04-25T09:18:40.584 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.617 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:726d7c01:::benchmark_data_smithi031_460268_object89:head# 2024-04-25T09:18:40.618 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.651 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72916a4f:::benchmark_data_smithi031_454752_object43:head# 2024-04-25T09:18:40.651 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.684 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72e5e7bc:::benchmark_data_smithi031_454752_object350:head# 2024-04-25T09:18:40.684 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.718 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72f2ccd2:::benchmark_data_smithi031_454752_object211:head# 2024-04-25T09:18:40.718 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.751 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:734a5a75:::benchmark_data_smithi031_460268_object2:head# 2024-04-25T09:18:40.751 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.785 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:737dcd92:::benchmark_data_smithi031_454752_object335:head# 2024-04-25T09:18:40.785 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.818 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:744eb9c4:::benchmark_data_smithi031_454752_object101:head# 2024-04-25T09:18:40.818 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.886 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:749ce32a:::benchmark_data_smithi031_460268_object142:head# 2024-04-25T09:18:40.886 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.936 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:74d52557:::benchmark_data_smithi031_460268_object56:head# 2024-04-25T09:18:40.936 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:40.969 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:750b59a6:::benchmark_data_smithi031_460268_object85:head# 2024-04-25T09:18:40.969 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.002 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75451e3a:::benchmark_data_smithi031_454752_object215:head# 2024-04-25T09:18:41.002 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.052 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75b87d24:::benchmark_data_smithi031_454752_object156:head# 2024-04-25T09:18:41.053 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.111 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76a44be7:::benchmark_data_smithi031_454752_object371:head# 2024-04-25T09:18:41.111 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.169 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76a5bace:::benchmark_data_smithi031_454752_object253:head# 2024-04-25T09:18:41.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.220 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:772eee67:::benchmark_data_smithi031_454752_object121:head# 2024-04-25T09:18:41.220 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.253 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:78512cf7:::benchmark_data_smithi031_460268_object139:head# 2024-04-25T09:18:41.253 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.329 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:78a2ff3a:::benchmark_data_smithi031_460268_object162:head# 2024-04-25T09:18:41.329 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.362 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79141f23:::benchmark_data_smithi031_460268_object78:head# 2024-04-25T09:18:41.362 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.396 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:798c5654:::benchmark_data_smithi031_454752_object204:head# 2024-04-25T09:18:41.396 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.429 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7995db24:::benchmark_data_smithi031_454752_object157:head# 2024-04-25T09:18:41.429 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.479 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79abc611:::benchmark_data_smithi031_454752_object164:head# 2024-04-25T09:18:41.480 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.512 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79d1490c:::benchmark_data_smithi031_454752_object138:head# 2024-04-25T09:18:41.513 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.546 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7adc14de:::benchmark_data_smithi031_460268_object43:head# 2024-04-25T09:18:41.546 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.580 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7b6edc17:::benchmark_data_smithi031_454752_object214:head# 2024-04-25T09:18:41.580 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.613 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7b93ca23:::benchmark_data_smithi031_454752_object55:head# 2024-04-25T09:18:41.613 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.647 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7ba8e7d6:::benchmark_data_smithi031_460268_object67:head# 2024-04-25T09:18:41.647 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.680 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bd5320a:::benchmark_data_smithi031_454752_object328:head# 2024-04-25T09:18:41.680 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.713 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bde86cb:::benchmark_data_smithi031_454752_object182:head# 2024-04-25T09:18:41.713 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.747 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bfed9be:::benchmark_data_smithi031_454752_object397:head# 2024-04-25T09:18:41.747 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.780 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7c002612:::benchmark_data_smithi031_460268_object110:head# 2024-04-25T09:18:41.780 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.813 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7c0b3015:::benchmark_data_smithi031_454752_object108:head# 2024-04-25T09:18:41.814 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.847 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7cadc342:::benchmark_data_smithi031_460268_object18:head# 2024-04-25T09:18:41.847 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.880 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7cb420f8:::benchmark_data_smithi031_454752_object89:head# 2024-04-25T09:18:41.881 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.914 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7d588f87:::benchmark_data_smithi031_454752_object376:head# 2024-04-25T09:18:41.914 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.947 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dbcb22e:::benchmark_data_smithi031_454752_object17:head# 2024-04-25T09:18:41.948 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dc6b3de:::benchmark_data_smithi031_454752_object187:head# 2024-04-25T09:18:41.981 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.014 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dec0b89:::benchmark_data_smithi031_454752_object149:head# 2024-04-25T09:18:42.015 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.048 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dfe5ddd:::benchmark_data_smithi031_454752_object79:head# 2024-04-25T09:18:42.048 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.081 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7ea5a1e2:::benchmark_data_smithi031_454752_object360:head# 2024-04-25T09:18:42.081 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.114 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7eeb5bde:::benchmark_data_smithi031_454752_object186:head# 2024-04-25T09:18:42.115 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.148 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7fb12f34:::benchmark_data_smithi031_454752_object69:head# 2024-04-25T09:18:42.148 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.198 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:800bce12:::benchmark_data_smithi031_460268_object118:head# 2024-04-25T09:18:42.198 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.232 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:80100b56:::benchmark_data_smithi031_460268_object22:head# 2024-04-25T09:18:42.232 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.265 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8187e1d1:::benchmark_data_smithi031_460268_object187:head# 2024-04-25T09:18:42.265 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:81a2bad0:::benchmark_data_smithi031_460268_object23:head# 2024-04-25T09:18:42.307 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.341 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:822ea472:::benchmark_data_smithi031_460268_object190:head# 2024-04-25T09:18:42.341 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.374 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:825fce9d:::benchmark_data_smithi031_454752_object9:head# 2024-04-25T09:18:42.374 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.416 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:82fe0341:::benchmark_data_smithi031_454752_object167:head# 2024-04-25T09:18:42.416 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.457 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:830b3926:::benchmark_data_smithi031_454752_object175:head# 2024-04-25T09:18:42.458 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.499 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:839ff2c1:::benchmark_data_smithi031_454752_object178:head# 2024-04-25T09:18:42.499 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.532 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:83bed9ab:::benchmark_data_smithi031_454752_object192:head# 2024-04-25T09:18:42.533 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.566 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:84bae7de:::benchmark_data_smithi031_454752_object19:head# 2024-04-25T09:18:42.566 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.600 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8514352c:::benchmark_data_smithi031_454752_object84:head# 2024-04-25T09:18:42.600 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.633 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:85dc3ddd:::benchmark_data_smithi031_460268_object92:head# 2024-04-25T09:18:42.633 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.666 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:85e422cb:::benchmark_data_smithi031_454752_object183:head# 2024-04-25T09:18:42.667 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.700 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8696227e:::benchmark_data_smithi031_454752_object304:head# 2024-04-25T09:18:42.700 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.733 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:869e69b7:::benchmark_data_smithi031_460268_object189:head# 2024-04-25T09:18:42.733 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.767 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:86aa36c1:::benchmark_data_smithi031_454752_object179:head# 2024-04-25T09:18:42.767 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.800 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:878ba262:::benchmark_data_smithi031_454752_object281:head# 2024-04-25T09:18:42.800 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.834 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:885c852e:::benchmark_data_smithi031_460268_object186:head# 2024-04-25T09:18:42.834 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.892 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:88d66484:::benchmark_data_smithi031_460268_object35:head# 2024-04-25T09:18:42.892 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.943 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:891fd6e5:::benchmark_data_smithi031_454752_object28:head# 2024-04-25T09:18:42.943 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:42.976 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:893b4bab:::benchmark_data_smithi031_460268_object113:head# 2024-04-25T09:18:42.976 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.009 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8941388a:::benchmark_data_smithi031_454752_object87:head# 2024-04-25T09:18:43.010 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.052 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:89e1ebba:::benchmark_data_smithi031_460268_object154:head# 2024-04-25T09:18:43.052 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.127 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8a4773c4:::benchmark_data_smithi031_454752_object103:head# 2024-04-25T09:18:43.127 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.185 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8aae6d43:::benchmark_data_smithi031_460268_object25:head# 2024-04-25T09:18:43.185 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.236 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8abbf253:::benchmark_data_smithi031_454752_object97:head# 2024-04-25T09:18:43.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.269 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8adf42cb:::benchmark_data_smithi031_454752_object180:head# 2024-04-25T09:18:43.269 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.302 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8ae888df:::benchmark_data_smithi031_454752_object382:head# 2024-04-25T09:18:43.302 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.336 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8b291374:::benchmark_data_smithi031_454752_object273:head# 2024-04-25T09:18:43.336 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.369 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8b621052:::benchmark_data_smithi031_454752_object368:head# 2024-04-25T09:18:43.369 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.403 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8c90b5b1:::benchmark_data_smithi031_454752_object161:head# 2024-04-25T09:18:43.403 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.436 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8d29fb8d:::benchmark_data_smithi031_454752_object327:head# 2024-04-25T09:18:43.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.470 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8d40e291:::benchmark_data_smithi031_460268_object120:head# 2024-04-25T09:18:43.470 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.487 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8db4e958:::benchmark_data_smithi031_454752_object385:head# 2024-04-25T09:18:43.487 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.504 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8eb4eb50:::benchmark_data_smithi031_460268_object8:head# 2024-04-25T09:18:43.504 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.521 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8f54b87d:::benchmark_data_smithi031_460268_object13:head# 2024-04-25T09:18:43.521 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.537 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8fa26b27:::benchmark_data_smithi031_454752_object256:head# 2024-04-25T09:18:43.537 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.554 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:90588a1a:::benchmark_data_smithi031_454752_object378:head# 2024-04-25T09:18:43.554 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.571 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:907e8918:::benchmark_data_smithi031_454752_object266:head# 2024-04-25T09:18:43.571 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.587 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:924cae69:::benchmark_data_smithi031_460268_object12:head# 2024-04-25T09:18:43.587 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.604 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9277f059:::benchmark_data_smithi031_454752_object2:head# 2024-04-25T09:18:43.604 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.621 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:933ca87a:::benchmark_data_smithi031_460268_object181:head# 2024-04-25T09:18:43.621 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.637 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:93d7997f:::benchmark_data_smithi031_460268_object6:head# 2024-04-25T09:18:43.637 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.654 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:93ded35d:::benchmark_data_smithi031_460268_object59:head# 2024-04-25T09:18:43.654 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.671 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9415aee4:::benchmark_data_smithi031_460268_object173:head# 2024-04-25T09:18:43.671 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.687 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:946e6356:::benchmark_data_smithi031_454752_object338:head# 2024-04-25T09:18:43.688 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.704 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:947d7692:::benchmark_data_smithi031_460268_object39:head# 2024-04-25T09:18:43.705 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.721 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9586170f:::benchmark_data_smithi031_454752_object13:head# 2024-04-25T09:18:43.721 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:95c08ad2:::benchmark_data_smithi031_454752_object210:head# 2024-04-25T09:18:43.738 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.750 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:95cd8cdf:::benchmark_data_smithi031_454752_object383:head# 2024-04-25T09:18:43.750 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.758 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9609476c:::benchmark_data_smithi031_454752_object114:head# 2024-04-25T09:18:43.758 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.766 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:967bc283:::benchmark_data_smithi031_454752_object231:head# 2024-04-25T09:18:43.767 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.775 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96801ae8:::benchmark_data_smithi031_454752_object207:head# 2024-04-25T09:18:43.775 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.783 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96a964d2:::benchmark_data_smithi031_454752_object38:head# 2024-04-25T09:18:43.783 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.791 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96aa787d:::benchmark_data_smithi031_454752_object322:head# 2024-04-25T09:18:43.792 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.800 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96d63edc:::benchmark_data_smithi031_454752_object352:head# 2024-04-25T09:18:43.800 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.808 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:97138c7c:::benchmark_data_smithi031_460268_object179:head# 2024-04-25T09:18:43.808 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.817 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:98e8b5bf:::benchmark_data_smithi031_454752_object216:head# 2024-04-25T09:18:43.817 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.825 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:99e3add0:::benchmark_data_smithi031_454752_object5:head# 2024-04-25T09:18:43.825 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.833 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a07c74e:::benchmark_data_smithi031_454752_object252:head# 2024-04-25T09:18:43.833 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.842 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a72fd5f:::benchmark_data_smithi031_454752_object297:head# 2024-04-25T09:18:43.842 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.850 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9a9f4dbd:::benchmark_data_smithi031_454752_object321:head# 2024-04-25T09:18:43.850 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.858 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9b86cf51:::benchmark_data_smithi031_454752_object225:head# 2024-04-25T09:18:43.858 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.867 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9be87125:::benchmark_data_smithi031_454752_object116:head# 2024-04-25T09:18:43.867 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.875 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9bf0e6c9:::benchmark_data_smithi031_454752_object47:head# 2024-04-25T09:18:43.875 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.884 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9c836d36:::benchmark_data_smithi031_454752_object44:head# 2024-04-25T09:18:43.884 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.908 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9cb1d2e2:::benchmark_data_smithi031_454752_object362:head# 2024-04-25T09:18:43.909 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.917 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9cbe41c2:::benchmark_data_smithi031_454752_object374:head# 2024-04-25T09:18:43.918 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.925 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9ce31ece:::benchmark_data_smithi031_454752_object308:head# 2024-04-25T09:18:43.926 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.934 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9df3f789:::benchmark_data_smithi031_454752_object394:head# 2024-04-25T09:18:43.934 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.942 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9e69968b:::benchmark_data_smithi031_460268_object126:head# 2024-04-25T09:18:43.942 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.950 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9e952085:::benchmark_data_smithi031_454752_object154:head# 2024-04-25T09:18:43.950 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.959 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9e95fdb1:::benchmark_data_smithi031_460268_object14:head# 2024-04-25T09:18:43.959 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:43.967 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9f2a7ece:::benchmark_data_smithi031_454752_object251:head# 2024-04-25T09:18:43.967 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.000 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a114ad36:::benchmark_data_smithi031_454752_object70:head# 2024-04-25T09:18:44.000 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.009 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a1535204:::benchmark_data_smithi031_460268_object71:head# 2024-04-25T09:18:44.009 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.017 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a2b210c5:::benchmark_data_smithi031_460268_object147:head# 2024-04-25T09:18:44.017 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a419031b:::benchmark_data_smithi031_454752_object81:head# 2024-04-25T09:18:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.034 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a45f5752:::benchmark_data_smithi031_454752_object39:head# 2024-04-25T09:18:44.034 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.042 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a46e3cab:::benchmark_data_smithi031_454752_object193:head# 2024-04-25T09:18:44.042 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.050 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a46e59f0:::benchmark_data_smithi031_454752_object130:head# 2024-04-25T09:18:44.051 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.059 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a47669cd:::benchmark_data_smithi031_460268_object82:head# 2024-04-25T09:18:44.059 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.067 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a4e9c0ae:::benchmark_data_smithi031_460268_object163:head# 2024-04-25T09:18:44.067 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.075 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a6bdfc76:::benchmark_data_smithi031_460268_object66:head# 2024-04-25T09:18:44.075 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.084 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a75374ab:::benchmark_data_smithi031_454752_object190:head# 2024-04-25T09:18:44.084 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a7706eea:::benchmark_data_smithi031_460268_object16:head# 2024-04-25T09:18:44.092 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.101 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8271a67:::benchmark_data_smithi031_454752_object33:head# 2024-04-25T09:18:44.101 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.109 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a853f81a:::benchmark_data_smithi031_454752_object379:head# 2024-04-25T09:18:44.109 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.117 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8c6bb46:::benchmark_data_smithi031_454752_object72:head# 2024-04-25T09:18:44.117 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.126 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a931c34a:::benchmark_data_smithi031_460268_object27:head# 2024-04-25T09:18:44.126 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.134 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a942e3ba:::benchmark_data_smithi031_454752_object74:head# 2024-04-25T09:18:44.134 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.143 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9870539:::benchmark_data_smithi031_460268_object83:head# 2024-04-25T09:18:44.143 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.150 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a98c4d7c:::benchmark_data_smithi031_454752_object37:head# 2024-04-25T09:18:44.151 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.159 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9e307d2:::benchmark_data_smithi031_454752_object318:head# 2024-04-25T09:18:44.159 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.167 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aab383a6:::benchmark_data_smithi031_454752_object112:head# 2024-04-25T09:18:44.167 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.176 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aaf58300:::benchmark_data_smithi031_454752_object208:head# 2024-04-25T09:18:44.176 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.184 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab0f7dc2:::benchmark_data_smithi031_454752_object119:head# 2024-04-25T09:18:44.184 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.192 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab1cf81e:::benchmark_data_smithi031_454752_object263:head# 2024-04-25T09:18:44.192 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.201 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab49749c:::benchmark_data_smithi031_454752_object359:head# 2024-04-25T09:18:44.201 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.209 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ac1d5d1f:::benchmark_data_smithi031_454752_object135:head# 2024-04-25T09:18:44.209 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.217 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ac7ea5bf:::benchmark_data_smithi031_454752_object217:head# 2024-04-25T09:18:44.217 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.226 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:acc3e569:::benchmark_data_smithi031_454752_object290:head# 2024-04-25T09:18:44.226 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.234 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ad398d0f:::benchmark_data_smithi031_454752_object125:head# 2024-04-25T09:18:44.234 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.242 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ae91e1ef:::benchmark_data_smithi031_454752_object115:head# 2024-04-25T09:18:44.242 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.251 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aea3dfa5:::benchmark_data_smithi031_454752_object388:head# 2024-04-25T09:18:44.251 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.268 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aec2dd45:::benchmark_data_smithi031_454752_object275:head# 2024-04-25T09:18:44.268 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.276 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aecd0faf:::benchmark_data_smithi031_460268_object156:head# 2024-04-25T09:18:44.276 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.284 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af668e34:::benchmark_data_smithi031_454752_object302:head# 2024-04-25T09:18:44.284 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.293 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af9a9573:::benchmark_data_smithi031_454752_object58:head# 2024-04-25T09:18:44.293 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.301 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:afcab8ff:::benchmark_data_smithi031_460268_object106:head# 2024-04-25T09:18:44.301 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.309 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b089bb4f:::benchmark_data_smithi031_454752_object218:head# 2024-04-25T09:18:44.310 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.318 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b1674de8:::benchmark_data_smithi031_454752_object194:head# 2024-04-25T09:18:44.318 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.326 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b2930ee9:::benchmark_data_smithi031_454752_object73:head# 2024-04-25T09:18:44.326 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.334 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b329d82c:::benchmark_data_smithi031_454752_object200:head# 2024-04-25T09:18:44.335 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.343 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b365eb28:::benchmark_data_smithi031_460268_object32:head# 2024-04-25T09:18:44.343 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.351 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b3c75192:::benchmark_data_smithi031_454752_object325:head# 2024-04-25T09:18:44.351 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.360 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b45e9911:::benchmark_data_smithi031_454752_object53:head# 2024-04-25T09:18:44.360 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.368 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b4f7e75f:::benchmark_data_smithi031_454752_object296:head# 2024-04-25T09:18:44.368 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.376 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b51eab52:::benchmark_data_smithi031_454752_object393:head# 2024-04-25T09:18:44.376 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.385 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b54079d2:::benchmark_data_smithi031_454752_object319:head# 2024-04-25T09:18:44.385 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.393 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b5427b66:::benchmark_data_smithi031_454752_object113:head# 2024-04-25T09:18:44.393 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.402 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b59f1134:::benchmark_data_smithi031_454752_object301:head# 2024-04-25T09:18:44.402 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.410 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b66cf944:::benchmark_data_smithi031_460268_object183:head# 2024-04-25T09:18:44.410 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.418 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b66ddae8:::benchmark_data_smithi031_454752_object206:head# 2024-04-25T09:18:44.418 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.426 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b6a31cad:::benchmark_data_smithi031_454752_object27:head# 2024-04-25T09:18:44.426 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.435 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b7073e4b:::benchmark_data_smithi031_454752_object126:head# 2024-04-25T09:18:44.435 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.443 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b713a907:::benchmark_data_smithi031_460268_object74:head# 2024-04-25T09:18:44.443 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.451 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b82823fa:::benchmark_data_smithi031_460268_object192:head# 2024-04-25T09:18:44.452 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.460 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b83f72b1:::benchmark_data_smithi031_454752_object162:head# 2024-04-25T09:18:44.460 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.468 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:18:44.468 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.493 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b8e53c75:::benchmark_data_smithi031_460268_object50:head# 2024-04-25T09:18:44.493 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.502 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b94a5f10:::benchmark_data_smithi031_454752_object348:head# 2024-04-25T09:18:44.502 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.510 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b9a0d9e0:::benchmark_data_smithi031_454752_object342:head# 2024-04-25T09:18:44.510 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.518 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b9e6e194:::benchmark_data_smithi031_460268_object34:head# 2024-04-25T09:18:44.518 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.527 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb052c00:::benchmark_data_smithi031_454752_object88:head# 2024-04-25T09:18:44.527 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.535 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb48e6d2:::benchmark_data_smithi031_454752_object241:head# 2024-04-25T09:18:44.535 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.543 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb57b519:::benchmark_data_smithi031_454752_object50:head# 2024-04-25T09:18:44.543 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.552 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb6af9e7:::benchmark_data_smithi031_454752_object373:head# 2024-04-25T09:18:44.552 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.560 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bbf3767d:::benchmark_data_smithi031_460268_object161:head# 2024-04-25T09:18:44.560 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.568 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bc520c76:::benchmark_data_smithi031_460268_object105:head# 2024-04-25T09:18:44.568 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.577 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bc8ac351:::benchmark_data_smithi031_460268_object36:head# 2024-04-25T09:18:44.577 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.585 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bca0f028:::benchmark_data_smithi031_454752_object20:head# 2024-04-25T09:18:44.585 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.593 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd00cd0f:::benchmark_data_smithi031_454752_object24:head# 2024-04-25T09:18:44.593 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.602 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd3a993f:::benchmark_data_smithi031_454752_object255:head# 2024-04-25T09:18:44.602 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.610 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd3fca90:::benchmark_data_smithi031_460268_object128:head# 2024-04-25T09:18:44.610 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.618 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bdd858f4:::benchmark_data_smithi031_454752_object228:head# 2024-04-25T09:18:44.619 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.627 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bdece741:::benchmark_data_smithi031_460268_object150:head# 2024-04-25T09:18:44.627 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.635 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bf17b80d:::benchmark_data_smithi031_460268_object101:head# 2024-04-25T09:18:44.635 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.643 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bfa77210:::benchmark_data_smithi031_454752_object67:head# 2024-04-25T09:18:44.644 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.652 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0619f0c:::benchmark_data_smithi031_454752_object247:head# 2024-04-25T09:18:44.652 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.660 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0b886c3:::benchmark_data_smithi031_454752_object25:head# 2024-04-25T09:18:44.660 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.668 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c1540ecc:::benchmark_data_smithi031_460268_object4:head# 2024-04-25T09:18:44.669 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.677 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c1bc9d65:::benchmark_data_smithi031_460268_object38:head# 2024-04-25T09:18:44.677 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.685 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c217c5ee:::benchmark_data_smithi031_460268_object115:head# 2024-04-25T09:18:44.685 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.694 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c229294b:::benchmark_data_smithi031_454752_object127:head# 2024-04-25T09:18:44.694 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.702 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c2647231:::benchmark_data_smithi031_460268_object174:head# 2024-04-25T09:18:44.702 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.710 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c2f8ad7b:::benchmark_data_smithi031_454752_object104:head# 2024-04-25T09:18:44.710 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.719 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c4d896cc:::benchmark_data_smithi031_460268_object153:head# 2024-04-25T09:18:44.719 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.727 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c4fe132d:::benchmark_data_smithi031_454752_object152:head# 2024-04-25T09:18:44.727 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.735 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c5aa9351:::benchmark_data_smithi031_460268_object119:head# 2024-04-25T09:18:44.735 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.744 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c5dc4341:::benchmark_data_smithi031_454752_object166:head# 2024-04-25T09:18:44.744 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.752 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c6367955:::benchmark_data_smithi031_454752_object144:head# 2024-04-25T09:18:44.752 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.760 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c664156b:::benchmark_data_smithi031_454752_object238:head# 2024-04-25T09:18:44.760 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.769 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c674ecb3:::benchmark_data_smithi031_460268_object117:head# 2024-04-25T09:18:44.769 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.777 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c78167ae:::benchmark_data_smithi031_454752_object85:head# 2024-04-25T09:18:44.777 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.785 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c79d3970:::benchmark_data_smithi031_454752_object221:head# 2024-04-25T09:18:44.786 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.794 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c86d58be:::benchmark_data_smithi031_460268_object65:head# 2024-04-25T09:18:44.794 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.802 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c8eb0827:::benchmark_data_smithi031_454752_object298:head# 2024-04-25T09:18:44.802 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.811 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c96d11d1:::benchmark_data_smithi031_460268_object30:head# 2024-04-25T09:18:44.811 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.819 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca1d51a2:::benchmark_data_smithi031_454752_object280:head# 2024-04-25T09:18:44.819 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.827 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca1f007e:::benchmark_data_smithi031_460268_object57:head# 2024-04-25T09:18:44.827 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.869 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cab67049:::benchmark_data_smithi031_454752_object91:head# 2024-04-25T09:18:44.869 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.894 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cad688fc:::benchmark_data_smithi031_454752_object351:head# 2024-04-25T09:18:44.894 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.902 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:caf98509:::benchmark_data_smithi031_454752_object196:head# 2024-04-25T09:18:44.902 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.911 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc58c178:::benchmark_data_smithi031_454752_object32:head# 2024-04-25T09:18:44.911 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.919 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc9c514e:::benchmark_data_smithi031_454752_object65:head# 2024-04-25T09:18:44.919 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.927 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cca29ffb:::benchmark_data_smithi031_454752_object80:head# 2024-04-25T09:18:44.927 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.936 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ccbd2f1f:::benchmark_data_smithi031_454752_object184:head# 2024-04-25T09:18:44.936 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.944 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ccc8b271:::benchmark_data_smithi031_460268_object33:head# 2024-04-25T09:18:44.944 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.952 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cce211e5:::benchmark_data_smithi031_460268_object42:head# 2024-04-25T09:18:44.952 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.961 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cd8693fa:::benchmark_data_smithi031_454752_object398:head# 2024-04-25T09:18:44.961 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.969 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ce45c38a:::benchmark_data_smithi031_460268_object93:head# 2024-04-25T09:18:44.969 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.977 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ceb870b0:::benchmark_data_smithi031_454752_object222:head# 2024-04-25T09:18:44.978 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.986 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d030878f:::benchmark_data_smithi031_460268_object45:head# 2024-04-25T09:18:44.986 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:44.994 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d093eb2e:::benchmark_data_smithi031_454752_object311:head# 2024-04-25T09:18:44.994 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.002 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d231d661:::benchmark_data_smithi031_454752_object177:head# 2024-04-25T09:18:45.003 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.011 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d2ae4d7f:::benchmark_data_smithi031_454752_object51:head# 2024-04-25T09:18:45.011 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.019 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d35c9cea:::benchmark_data_smithi031_454752_object317:head# 2024-04-25T09:18:45.019 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.028 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d40521f8:::benchmark_data_smithi031_460268_object52:head# 2024-04-25T09:18:45.028 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.036 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d4438560:::benchmark_data_smithi031_454752_object143:head# 2024-04-25T09:18:45.036 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.044 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d452fc99:::benchmark_data_smithi031_454752_object45:head# 2024-04-25T09:18:45.044 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.053 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d6165b5f:::benchmark_data_smithi031_460268_object7:head# 2024-04-25T09:18:45.053 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.078 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d6574e4a:::benchmark_data_smithi031_454752_object60:head# 2024-04-25T09:18:45.078 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.170 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d73435c2:::benchmark_data_smithi031_454752_object118:head# 2024-04-25T09:18:45.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.212 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75560e8:::benchmark_data_smithi031_454752_object128:head# 2024-04-25T09:18:45.212 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.236 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75a04bb:::benchmark_data_smithi031_454752_object98:head# 2024-04-25T09:18:45.236 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.245 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d75b7183:::benchmark_data_smithi031_454752_object230:head# 2024-04-25T09:18:45.245 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.253 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d7bf5e5b:::benchmark_data_smithi031_454752_object239:head# 2024-04-25T09:18:45.253 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.261 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d824c896:::benchmark_data_smithi031_454752_object286:head# 2024-04-25T09:18:45.261 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.269 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da211111:::benchmark_data_smithi031_454752_object159:head# 2024-04-25T09:18:45.269 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.278 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da25a4ff:::benchmark_data_smithi031_454752_object75:head# 2024-04-25T09:18:45.278 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.286 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da449e8d:::benchmark_data_smithi031_460268_object91:head# 2024-04-25T09:18:45.286 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.294 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:db18fece:::benchmark_data_smithi031_454752_object250:head# 2024-04-25T09:18:45.295 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dc4325f0:::benchmark_data_smithi031_454752_object131:head# 2024-04-25T09:18:45.303 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.311 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd2df7d0:::benchmark_data_smithi031_460268_object20:head# 2024-04-25T09:18:45.311 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.319 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd89516f:::benchmark_data_smithi031_460268_object112:head# 2024-04-25T09:18:45.320 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dda1d8b9:::benchmark_data_smithi031_460268_object28:head# 2024-04-25T09:18:45.328 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:deb7d977:::benchmark_data_smithi031_460268_object134:head# 2024-04-25T09:18:45.336 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.344 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dedcc3de:::benchmark_data_smithi031_454752_object148:head# 2024-04-25T09:18:45.345 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:deec6ff8:::benchmark_data_smithi031_454752_object224:head# 2024-04-25T09:18:45.361 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e0881e69:::benchmark_data_smithi031_454752_object293:head# 2024-04-25T09:18:45.378 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e1341170:::benchmark_data_smithi031_454752_object132:head# 2024-04-25T09:18:45.386 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.395 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e17ed360:::benchmark_data_smithi031_454752_object140:head# 2024-04-25T09:18:45.395 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.403 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3835ce2:::benchmark_data_smithi031_460268_object188:head# 2024-04-25T09:18:45.403 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.412 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3be378c:::benchmark_data_smithi031_454752_object188:head# 2024-04-25T09:18:45.412 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.420 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e3c37152:::benchmark_data_smithi031_454752_object155:head# 2024-04-25T09:18:45.420 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.428 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e431150e:::benchmark_data_smithi031_454752_object309:head# 2024-04-25T09:18:45.428 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e44b8a2c:::benchmark_data_smithi031_454752_object146:head# 2024-04-25T09:18:45.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.445 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e469a838:::benchmark_data_smithi031_460268_object63:head# 2024-04-25T09:18:45.445 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.453 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e4a3dd60:::benchmark_data_smithi031_454752_object341:head# 2024-04-25T09:18:45.453 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.461 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e53228e8:::benchmark_data_smithi031_454752_object364:head# 2024-04-25T09:18:45.462 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.470 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e53430a0:::benchmark_data_smithi031_454752_object336:head# 2024-04-25T09:18:45.470 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.478 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e5497360:::benchmark_data_smithi031_454752_object141:head# 2024-04-25T09:18:45.478 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.486 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e5ae028f:::benchmark_data_smithi031_454752_object254:head# 2024-04-25T09:18:45.487 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.495 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e653ea24:::benchmark_data_smithi031_460268_object51:head# 2024-04-25T09:18:45.495 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.503 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e72dd955:::benchmark_data_smithi031_454752_object105:head# 2024-04-25T09:18:45.503 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.511 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e75527e1:::benchmark_data_smithi031_454752_object365:head# 2024-04-25T09:18:45.512 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.520 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e7710ac5:::benchmark_data_smithi031_460268_object53:head# 2024-04-25T09:18:45.520 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.528 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e8ddccdd:::benchmark_data_smithi031_460268_object31:head# 2024-04-25T09:18:45.528 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.536 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e8e4cd11:::benchmark_data_smithi031_454752_object171:head# 2024-04-25T09:18:45.537 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.545 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e94a0572:::benchmark_data_smithi031_454752_object6:head# 2024-04-25T09:18:45.545 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.553 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e991363a:::benchmark_data_smithi031_454752_object52:head# 2024-04-25T09:18:45.553 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.562 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e9cfc042:::benchmark_data_smithi031_460268_object24:head# 2024-04-25T09:18:45.562 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.570 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ea07a4ce:::benchmark_data_smithi031_460268_object3:head# 2024-04-25T09:18:45.570 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.578 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ea70172b:::benchmark_data_smithi031_454752_object191:head# 2024-04-25T09:18:45.579 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.620 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:eaf3d89c:::benchmark_data_smithi031_454752_object358:head# 2024-04-25T09:18:45.620 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.637 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ec070df6:::benchmark_data_smithi031_454752_object234:head# 2024-04-25T09:18:45.637 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.654 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ec82b70c:::benchmark_data_smithi031_454752_object246:head# 2024-04-25T09:18:45.654 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.670 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ed17f3a5:::benchmark_data_smithi031_454752_object389:head# 2024-04-25T09:18:45.671 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.687 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:edd73bf9:::benchmark_data_smithi031_454752_object95:head# 2024-04-25T09:18:45.688 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.695 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ee0f64f0:::benchmark_data_smithi031_454752_object49:head# 2024-04-25T09:18:45.695 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.704 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ee2ac15d:::benchmark_data_smithi031_460268_object100:head# 2024-04-25T09:18:45.704 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.712 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f054ad11:::benchmark_data_smithi031_454752_object145:head# 2024-04-25T09:18:45.712 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.728 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f0cae848:::benchmark_data_smithi031_460268_object136:head# 2024-04-25T09:18:45.729 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.746 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f102181e:::benchmark_data_smithi031_454752_object288:head# 2024-04-25T09:18:45.746 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.762 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f155a6c1:::benchmark_data_smithi031_460268_object124:head# 2024-04-25T09:18:45.762 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.779 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f21d79de:::benchmark_data_smithi031_454752_object66:head# 2024-04-25T09:18:45.779 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.787 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f2b747ee:::benchmark_data_smithi031_454752_object314:head# 2024-04-25T09:18:45.787 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.795 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f308d551:::benchmark_data_smithi031_460268_object84:head# 2024-04-25T09:18:45.795 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.804 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f34b8ad2:::benchmark_data_smithi031_454752_object240:head# 2024-04-25T09:18:45.804 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.820 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4863048:::benchmark_data_smithi031_454752_object349:head# 2024-04-25T09:18:45.821 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.837 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4ab8147:::benchmark_data_smithi031_454752_object337:head# 2024-04-25T09:18:45.837 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.854 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f53a131c:::benchmark_data_smithi031_460268_object17:head# 2024-04-25T09:18:45.854 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.870 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f544082b:::benchmark_data_smithi031_460268_object81:head# 2024-04-25T09:18:45.870 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.887 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f547a4d2:::benchmark_data_smithi031_454752_object369:head# 2024-04-25T09:18:45.888 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.904 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f5b63e7c:::benchmark_data_smithi031_460268_object171:head# 2024-04-25T09:18:45.904 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.921 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6236fee:::benchmark_data_smithi031_454752_object289:head# 2024-04-25T09:18:45.921 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.937 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f635f136:::benchmark_data_smithi031_454752_object23:head# 2024-04-25T09:18:45.937 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.962 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6e629fe:::benchmark_data_smithi031_454752_object57:head# 2024-04-25T09:18:45.963 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.979 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f70fb177:::benchmark_data_smithi031_454752_object265:head# 2024-04-25T09:18:45.979 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:45.996 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f77401b8:::benchmark_data_smithi031_454752_object71:head# 2024-04-25T09:18:45.996 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.012 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f7954077:::benchmark_data_smithi031_460268_object145:head# 2024-04-25T09:18:46.012 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.029 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f870ff0e:::benchmark_data_smithi031_454752_object312:head# 2024-04-25T09:18:46.029 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.046 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f8855049:::benchmark_data_smithi031_460268_object94:head# 2024-04-25T09:18:46.046 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.062 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f90eb4af:::benchmark_data_smithi031_460268_object185:head# 2024-04-25T09:18:46.062 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.079 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f93bf106:::benchmark_data_smithi031_460268_object72:head# 2024-04-25T09:18:46.079 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.096 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fa3f28f3:::benchmark_data_smithi031_460268_object166:head# 2024-04-25T09:18:46.096 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.112 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:faefa441:::benchmark_data_smithi031_454752_object269:head# 2024-04-25T09:18:46.112 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.129 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb1204b7:::benchmark_data_smithi031_454752_object331:head# 2024-04-25T09:18:46.129 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.138 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb3355e2:::benchmark_data_smithi031_454752_object363:head# 2024-04-25T09:18:46.138 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.154 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb5c543a:::benchmark_data_smithi031_454752_object329:head# 2024-04-25T09:18:46.154 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.171 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fbfbcb4a:::benchmark_data_smithi031_460268_object104:head# 2024-04-25T09:18:46.171 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.179 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fcf4159f:::benchmark_data_smithi031_454752_object185:head# 2024-04-25T09:18:46.179 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.187 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fe5d917d:::benchmark_data_smithi031_460268_object182:head# 2024-04-25T09:18:46.187 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.196 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fe7c790a:::benchmark_data_smithi031_454752_object16:head# 2024-04-25T09:18:46.196 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.212 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ffeaaa0c:::benchmark_data_smithi031_454752_object139:head# 2024-04-25T09:18:46.213 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:18:46.229 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 225 info 1.0( v 225'407 (10'213,225'407] local-lis/les=24/25 n=595 ec=7/7 lis/c=24/24 les/c/f=25/25/0 sis=24) 2024-04-25T09:18:46.254 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T09:18:46.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:18:46.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:18:46.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:18:46.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:18:46.796 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:18:46.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:18:46.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:18:46.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:18:46.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:18:46.798 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:18:46.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:18:46.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:18:46.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:18:46.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:18:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:18:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:18:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:18:46.835 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:18:46.846+0000 7fb1be96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:46.855 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:18:46.866+0000 7fb1be96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:46.856 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:18:46.870+0000 7fb1be96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:47.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:18:47.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:18:47.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:18:47.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:18:47.270 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:18:47.270 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:18:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:18:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:47.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:47.417 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:18:47.430+0000 7fb1be96a7c0 -1 Falling back to public interface 2024-04-25T09:18:47.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:48.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:48.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:49.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:50.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:50.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:50.208 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:18:50.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:18:50.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:50.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:50.262 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:18:50.274+0000 7fb1be96a7c0 -1 osd.0 229 log_to_monitors true 2024-04-25T09:18:50.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:51.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:51.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:51.699 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:18:51.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:18:51.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:51.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:52.130 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 232 up_thru 232 down_at 230 last_clean_interval [24,229) [v2:127.0.0.1:6802/4289732269,v1:127.0.0.1:6803/4289732269] [v2:127.0.0.1:6804/4289732269,v1:127.0.0.1:6805/4289732269] exists,up d91c4f80-3723-41df-b7cb-c0ba7c928b9a 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:18:52.131 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:18:52.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:18:52.132 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:18:52.132 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:18:52.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:18:52.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:18:52.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:18:52.235 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:18:52.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:18:52.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:18:52.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:18:52.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:18:52.236 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:18:52.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:18:52.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:18:52.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:52.649 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:18:52.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=996432412675 2024-04-25T09:18:52.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 996432412675 2024-04-25T09:18:52.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-996432412675' 2024-04-25T09:18:52.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:52.828 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-996432412675 2024-04-25T09:18:52.828 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:52.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:18:52.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:52.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-996432412675 2024-04-25T09:18:52.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=996432412675 2024-04-25T09:18:52.832 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 996432412675 2024-04-25T09:18:52.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 996432412675' 2024-04-25T09:18:52.832 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:53.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 996432412675 2024-04-25T09:18:53.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:54.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:18:54.324 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:54.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 996432412675 2024-04-25T09:18:54.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:55.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-25T09:18:55.809 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:56.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 996432412675 -lt 996432412675 2024-04-25T09:18:56.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:18:56.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:56.251 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:56.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:18:56.827 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:18:57.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:18:57.248 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:18:57.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:57.249 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:57.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:57.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:57.930 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:57.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:57.931 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:57.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:57.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:57.932 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:57.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:57.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:57.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:57.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:57.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:57.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:18:57.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:18:57.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:57.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:57.951 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:57.951 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:18:57.952 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:57.953 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:57.954 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:57.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:57.955 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:57.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:57.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:57.956 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:57.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:57.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:57.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:57.957 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:57.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:57.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:18:57.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:18:57.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:57.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:57.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:57.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:18:57.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:57.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:57.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:18:57.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:18:57.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:57.962 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:57.962 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.443986 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:18:57.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:18:57.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:18:57.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:18:58.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:18:58.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:58.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:58.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:58.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:58.425 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:58.425 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:18:58.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:18:58.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:18:58.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:18:58.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:18:58.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:18:58.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:18:58.637 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:18:58.638 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:18:58.638 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:18:58.638 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:18:58.638 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:58.638 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:58.639 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:58.639 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.443986/ceph-mon.a.asok 2024-04-25T09:18:58.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:18:58.639 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.443986/ceph-mon.a.asok config get fsid 2024-04-25T09:18:58.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:18:58.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:18:58.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:18:58.758 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:18:58.760 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:58.760 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:58.760 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:58.760 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.443986/ceph-mon.a.asok 2024-04-25T09:18:58.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:18:58.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.443986/ceph-mon.a.asok config get mon_host 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:18:58.869 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:18:59.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:18:59.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:59.793 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:59.794 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:59.794 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:59.794 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:59.794 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:18:59.795 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:18:59.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:18:59.851 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:59.852 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:18:59.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:18:59.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:18:59.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:18:59.854 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:18:59.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e17a2f7-7821-4592-a394-a058baf89f2d 2024-04-25T09:18:59.855 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 7e17a2f7-7821-4592-a394-a058baf89f2d 2024-04-25T09:18:59.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7e17a2f7-7821-4592-a394-a058baf89f2d' 2024-04-25T09:18:59.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:18:59.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADICpmChT6NBAA9hv3spNJBtTt9/nyLQHBOw== 2024-04-25T09:18:59.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADICpmChT6NBAA9hv3spNJBtTt9/nyLQHBOw=="}' 2024-04-25T09:18:59.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e17a2f7-7821-4592-a394-a058baf89f2d -i td/pg-split-merge/0/new.json 2024-04-25T09:19:00.164 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:19:00.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:19:00.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADICpmChT6NBAA9hv3spNJBtTt9/nyLQHBOw== --osd-uuid 7e17a2f7-7821-4592-a394-a058baf89f2d 2024-04-25T09:19:00.211 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.222+0000 7fc89fec17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:00.232 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.246+0000 7fc89fec17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:00.233 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.246+0000 7fc89fec17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:00.234 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.246+0000 7fc89fec17c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:19:00.234 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.246+0000 7fc89fec17c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:19:00.234 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.246+0000 7fc89fec17c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:19:00.272 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:00.286+0000 7fc89fec17c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:19:06.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:19:06.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:19:06.488 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:19:06.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:19:06.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:19:07.065 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:19:07.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:19:07.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:19:07.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:19:07.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:19:07.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:19:07.094 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:07.106+0000 7f29c027d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:07.097 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:07.110+0000 7f29c027d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:07.099 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:07.110+0000 7f29c027d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:07.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:19:07.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:19:07.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:19:07.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:19:07.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:19:07.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:07.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:19:07.494 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:19:07.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:07.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:07.665 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:07.678+0000 7f29c027d7c0 -1 Falling back to public interface 2024-04-25T09:19:07.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:08.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:08.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:08.965 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:19:08.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:08.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:08.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:09.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:10.211 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:10.222+0000 7f29c027d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:19:10.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:10.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:10.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:10.397 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:19:10.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:10.397 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:10.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:11.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:11.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:11.933 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:19:11.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:11.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:11.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:12.436 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:13.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:13.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:13.438 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:19:13.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:19:13.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:13.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:13.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:14.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:14.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:14.916 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:19:14.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:19:14.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:14.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:15.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:16.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:16.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:16.358 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:19:16.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:19:16.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:16.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:16.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:17.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:17.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:17.931 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:19:17.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:19:17.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:17.932 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:18.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:19.404 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:19:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:19:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:19.965 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:20.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:20.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:20.967 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:19:20.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:19:20.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:20.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:21.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:22.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:22.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:22.410 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:19:22.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:19:22.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:22.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:23.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:24.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:24.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:24.029 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:19:24.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:19:24.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:24.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:24.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:25.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:25.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:25.499 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:19:25.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:19:25.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:25.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:26.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:27.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:19:27.305 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:19:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:27.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:28.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:29.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:29.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:29.325 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:19:29.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:19:29.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:29.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:30.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:31.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:19:31.383 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:19:31.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:31.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:32.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:33.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:33.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:33.336 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:19:33.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:19:33.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:33.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:33.817 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/955719249,v1:127.0.0.1:6803/955719249] [v2:127.0.0.1:6804/955719249,v1:127.0.0.1:6805/955719249] exists,up 7e17a2f7-7821-4592-a394-a058baf89f2d 2024-04-25T09:19:33.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:19:33.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:19:33.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:19:33.819 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-04-25T09:19:34.344 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'foo' already exists 2024-04-25T09:19:34.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-04-25T09:19:34.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:19:34.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:19:34.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:19:34.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:19:34.362 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:19:34.362 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:19:34.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:19:34.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:19:34.363 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:19:34.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:19:34.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:19:34.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:19:34.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:19:34.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:19:34.463 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:19:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:19:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:19:34.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:19:34.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:19:35.126 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836486 2024-04-25T09:19:35.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836486 2024-04-25T09:19:35.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836486' 2024-04-25T09:19:35.127 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:19:35.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836486 2024-04-25T09:19:35.127 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:19:35.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:19:35.132 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-04-25T09:19:35.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:19:35.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836486 2024-04-25T09:19:35.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-04-25T09:19:35.133 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836486 2024-04-25T09:19:35.133 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:19:35.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836486 2024-04-25T09:19:35.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:19:36.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:19:36.571 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:19:37.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836486 2024-04-25T09:19:37.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:19:37.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:19:37.031 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:19:37.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:19:37.601 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:19:38.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:19:38.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:19:38.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:19:38.026 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:19:38.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:19:38.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:19:38.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:19:38.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:19:38.617 INFO:tasks.workunit.client.0.smithi031.stdout:hints = 1 2024-04-25T09:19:38.617 INFO:tasks.workunit.client.0.smithi031.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:19:38.617 INFO:tasks.workunit.client.0.smithi031.stdout:Object prefix: benchmark_data_smithi031_467097 2024-04-25T09:19:38.617 INFO:tasks.workunit.client.0.smithi031.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:19:38.618 INFO:tasks.workunit.client.0.smithi031.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:19:39.617 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16 129 113 0.11034 0.110352 0.100562 0.131741 2024-04-25T09:19:40.617 INFO:tasks.workunit.client.0.smithi031.stdout: 2 16 288 272 0.132795 0.155273 0.100956 0.115485 2024-04-25T09:19:41.617 INFO:tasks.workunit.client.0.smithi031.stdout: 3 16 417 401 0.130516 0.125977 0.10086 0.117052 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Total time run: 3.07961 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Total writes made: 417 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Write size: 1024 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Object size: 1024 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Bandwidth (MB/sec): 0.132233 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Bandwidth: 0.0228051 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Max bandwidth (MB/sec): 0.155273 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Min bandwidth (MB/sec): 0.110352 2024-04-25T09:19:42.618 INFO:tasks.workunit.client.0.smithi031.stdout:Average IOPS: 135 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev IOPS: 23.3524 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Max IOPS: 159 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Min IOPS: 113 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Average Latency(s): 0.116387 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Stddev Latency(s): 0.0294645 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Max latency(s): 0.199718 2024-04-25T09:19:42.619 INFO:tasks.workunit.client.0.smithi031.stdout:Min latency(s): 0.0915267 2024-04-25T09:19:42.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:19:42.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:42.696 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:42.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:42.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:42.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:44.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:44.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:19:44.159 INFO:tasks.workunit.client.0.smithi031.stderr:Exporting 1.0 info 1.0( v 9'418 (0'0,9'418] local-lis/les=7/8 n=418 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:19:44.172 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:006fbae5:::benchmark_data_smithi031_467097_object305:head# 2024-04-25T09:19:44.172 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00dc8f94:::benchmark_data_smithi031_467097_object231:head# 2024-04-25T09:19:44.174 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:00e4b61e:::benchmark_data_smithi031_467097_object192:head# 2024-04-25T09:19:44.175 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:01853150:::benchmark_data_smithi031_467097_object123:head# 2024-04-25T09:19:44.176 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:02029128:::benchmark_data_smithi031_467097_object33:head# 2024-04-25T09:19:44.176 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:035e3271:::benchmark_data_smithi031_467097_object336:head# 2024-04-25T09:19:44.177 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:042d88ac:::benchmark_data_smithi031_467097_object175:head# 2024-04-25T09:19:44.178 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:050ee465:::benchmark_data_smithi031_467097_object71:head# 2024-04-25T09:19:44.178 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:05563abd:::benchmark_data_smithi031_467097_object193:head# 2024-04-25T09:19:44.179 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0650a180:::benchmark_data_smithi031_467097_object345:head# 2024-04-25T09:19:44.185 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:06e12f21:::benchmark_data_smithi031_467097_object128:head# 2024-04-25T09:19:44.186 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0735b904:::benchmark_data_smithi031_467097_object311:head# 2024-04-25T09:19:44.186 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:07baf8c0:::benchmark_data_smithi031_467097_object165:head# 2024-04-25T09:19:44.187 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:07cbb58b:::benchmark_data_smithi031_467097_object249:head# 2024-04-25T09:19:44.187 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:08aa4630:::benchmark_data_smithi031_467097_object257:head# 2024-04-25T09:19:44.188 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:092a861b:::benchmark_data_smithi031_467097_object191:head# 2024-04-25T09:19:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:092fb541:::benchmark_data_smithi031_467097_object139:head# 2024-04-25T09:19:44.189 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:09c6b1c6:::benchmark_data_smithi031_467097_object359:head# 2024-04-25T09:19:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0b4938ac:::benchmark_data_smithi031_467097_object171:head# 2024-04-25T09:19:44.190 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ba7118e:::benchmark_data_smithi031_467097_object59:head# 2024-04-25T09:19:44.198 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0cc5814b:::benchmark_data_smithi031_467097_object153:head# 2024-04-25T09:19:44.198 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ccd0c3e:::benchmark_data_smithi031_467097_object389:head# 2024-04-25T09:19:44.200 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0d1840e3:::benchmark_data_smithi031_467097_object272:head# 2024-04-25T09:19:44.200 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0ded6035:::benchmark_data_smithi031_467097_object393:head# 2024-04-25T09:19:44.203 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0e6631d3:::benchmark_data_smithi031_467097_object400:head# 2024-04-25T09:19:44.203 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:0e9d9805:::benchmark_data_smithi031_467097_object18:head# 2024-04-25T09:19:44.204 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1088b06d:::benchmark_data_smithi031_467097_object239:head# 2024-04-25T09:19:44.204 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:123cbe44:::benchmark_data_smithi031_467097_object12:head# 2024-04-25T09:19:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:124e586f:::benchmark_data_smithi031_467097_object313:head# 2024-04-25T09:19:44.205 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:12c4d666:::benchmark_data_smithi031_467097_object214:head# 2024-04-25T09:19:44.206 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:12f7f36c:::benchmark_data_smithi031_467097_object236:head# 2024-04-25T09:19:44.207 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:13ac91cc:::benchmark_data_smithi031_467097_object353:head# 2024-04-25T09:19:44.207 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:14dc840c:::benchmark_data_smithi031_467097_object181:head# 2024-04-25T09:19:44.208 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1572997c:::benchmark_data_smithi031_467097_object166:head# 2024-04-25T09:19:44.208 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:158037c9:::benchmark_data_smithi031_467097_object333:head# 2024-04-25T09:19:44.209 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1604cc8a:::benchmark_data_smithi031_467097_object370:head# 2024-04-25T09:19:44.209 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1638a65d:::benchmark_data_smithi031_467097_object183:head# 2024-04-25T09:19:44.210 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:16615529:::benchmark_data_smithi031_467097_object297:head# 2024-04-25T09:19:44.211 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:187801dc:::benchmark_data_smithi031_467097_object187:head# 2024-04-25T09:19:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18919c57:::benchmark_data_smithi031_467097_object230:head# 2024-04-25T09:19:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:18bd38b5:::benchmark_data_smithi031_467097_object49:head# 2024-04-25T09:19:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:19a5313b:::benchmark_data_smithi031_467097_object134:head# 2024-04-25T09:19:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1a967e52:::benchmark_data_smithi031_467097_object385:head# 2024-04-25T09:19:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1b51d208:::benchmark_data_smithi031_467097_object55:head# 2024-04-25T09:19:44.214 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1c76729f:::benchmark_data_smithi031_467097_object4:head# 2024-04-25T09:19:44.215 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1c8202ce:::benchmark_data_smithi031_467097_object241:head# 2024-04-25T09:19:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1d01f931:::benchmark_data_smithi031_467097_object237:head# 2024-04-25T09:19:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1d3a89fe:::benchmark_data_smithi031_467097_object113:head# 2024-04-25T09:19:44.216 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1df332eb:::benchmark_data_smithi031_467097_object73:head# 2024-04-25T09:19:44.217 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1e97ad6f:::benchmark_data_smithi031_467097_object394:head# 2024-04-25T09:19:44.217 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1f08b615:::benchmark_data_smithi031_467097_object224:head# 2024-04-25T09:19:44.217 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:1f197bc1:::benchmark_data_smithi031_467097_object62:head# 2024-04-25T09:19:44.218 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:212a5056:::benchmark_data_smithi031_467097_object117:head# 2024-04-25T09:19:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2220816f:::benchmark_data_smithi031_467097_object390:head# 2024-04-25T09:19:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:22326f7a:::benchmark_data_smithi031_467097_object36:head# 2024-04-25T09:19:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:22c06904:::benchmark_data_smithi031_467097_object235:head# 2024-04-25T09:19:44.220 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:252b1027:::benchmark_data_smithi031_467097_object411:head# 2024-04-25T09:19:44.220 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:25a5f6e6:::benchmark_data_smithi031_467097_object293:head# 2024-04-25T09:19:44.221 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:25b10ee1:::benchmark_data_smithi031_467097_object188:head# 2024-04-25T09:19:44.221 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:269bc0eb:::benchmark_data_smithi031_467097_object172:head# 2024-04-25T09:19:44.222 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:26a4dc22:::benchmark_data_smithi031_467097_object270:head# 2024-04-25T09:19:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:26df22c7:::benchmark_data_smithi031_467097_object369:head# 2024-04-25T09:19:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2709f079:::benchmark_data_smithi031_467097_object396:head# 2024-04-25T09:19:44.224 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:278b7215:::benchmark_data_smithi031_467097_object154:head# 2024-04-25T09:19:44.225 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:279e379a:::benchmark_data_smithi031_467097_object42:head# 2024-04-25T09:19:44.226 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:284bebcd:::benchmark_data_smithi031_467097_object319:head# 2024-04-25T09:19:44.226 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:296b8d2f:::benchmark_data_smithi031_467097_object383:head# 2024-04-25T09:19:44.227 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2a3369ea:::benchmark_data_smithi031_467097_object72:head# 2024-04-25T09:19:44.227 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2abcfdaf:::benchmark_data_smithi031_467097_object138:head# 2024-04-25T09:19:44.228 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2bbcf18f:::benchmark_data_smithi031_467097_object317:head# 2024-04-25T09:19:44.228 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2ddd5618:::benchmark_data_smithi031_467097_object283:head# 2024-04-25T09:19:44.229 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2e171fa8:::benchmark_data_smithi031_467097_object289:head# 2024-04-25T09:19:44.229 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2e3b47e8:::benchmark_data_smithi031_467097_object148:head# 2024-04-25T09:19:44.230 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2e3d6ee6:::benchmark_data_smithi031_467097_object295:head# 2024-04-25T09:19:44.230 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:2fcfc71f:::benchmark_data_smithi031_467097_object105:head# 2024-04-25T09:19:44.231 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3074c2ca:::benchmark_data_smithi031_467097_object268:head# 2024-04-25T09:19:44.231 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:30d72a55:::benchmark_data_smithi031_467097_object150:head# 2024-04-25T09:19:44.232 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:31f63d35:::benchmark_data_smithi031_467097_object119:head# 2024-04-25T09:19:44.232 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:32e46cf4:::benchmark_data_smithi031_467097_object278:head# 2024-04-25T09:19:44.233 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3553be04:::benchmark_data_smithi031_467097_object233:head# 2024-04-25T09:19:44.233 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:358d4ec2:::benchmark_data_smithi031_467097_object29:head# 2024-04-25T09:19:44.235 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:35d253e3:::benchmark_data_smithi031_467097_object38:head# 2024-04-25T09:19:44.235 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:36116e42:::benchmark_data_smithi031_467097_object149:head# 2024-04-25T09:19:44.235 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:362baf9a:::benchmark_data_smithi031_467097_object415:head# 2024-04-25T09:19:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3895cf62:::benchmark_data_smithi031_467097_object306:head# 2024-04-25T09:19:44.236 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:391dff5e:::benchmark_data_smithi031_467097_object16:head# 2024-04-25T09:19:44.237 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3924b3fd:::benchmark_data_smithi031_467097_object22:head# 2024-04-25T09:19:44.237 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:399c8e05:::benchmark_data_smithi031_467097_object342:head# 2024-04-25T09:19:44.238 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3a899b62:::benchmark_data_smithi031_467097_object56:head# 2024-04-25T09:19:44.238 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3b262498:::benchmark_data_smithi031_467097_object213:head# 2024-04-25T09:19:44.239 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3b64e165:::benchmark_data_smithi031_467097_object211:head# 2024-04-25T09:19:44.239 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3c71360f:::benchmark_data_smithi031_467097_object169:head# 2024-04-25T09:19:44.240 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3d086d1b:::benchmark_data_smithi031_467097_object142:head# 2024-04-25T09:19:44.240 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3e8ccd0d:::benchmark_data_smithi031_467097_object367:head# 2024-04-25T09:19:44.241 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3eecfee6:::benchmark_data_smithi031_467097_object145:head# 2024-04-25T09:19:44.242 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:3f13318f:::benchmark_data_smithi031_467097_object382:head# 2024-04-25T09:19:44.242 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:40762230:::benchmark_data_smithi031_467097_object271:head# 2024-04-25T09:19:44.243 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:41ba4b29:::benchmark_data_smithi031_467097_object299:head# 2024-04-25T09:19:44.243 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:42496b2f:::benchmark_data_smithi031_467097_object410:head# 2024-04-25T09:19:44.244 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:427f0ab0:::benchmark_data_smithi031_467097_object218:head# 2024-04-25T09:19:44.244 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:43924006:::benchmark_data_smithi031_467097_object248:head# 2024-04-25T09:19:44.245 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:458620e0:::benchmark_data_smithi031_467097_object40:head# 2024-04-25T09:19:44.245 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:46af7308:::benchmark_data_smithi031_467097_object143:head# 2024-04-25T09:19:44.246 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:46b81f28:::benchmark_data_smithi031_467097_object184:head# 2024-04-25T09:19:44.246 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:47173b92:::benchmark_data_smithi031_467097_object206:head# 2024-04-25T09:19:44.247 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:498d65aa:::benchmark_data_smithi031_467097_object103:head# 2024-04-25T09:19:44.248 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:499fa119:::benchmark_data_smithi031_467097_object348:head# 2024-04-25T09:19:44.249 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4b1439e9:::benchmark_data_smithi031_467097_object156:head# 2024-04-25T09:19:44.249 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4bd3331c:::benchmark_data_smithi031_467097_object277:head# 2024-04-25T09:19:44.250 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4d367541:::benchmark_data_smithi031_467097_object137:head# 2024-04-25T09:19:44.250 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4d6e4159:::benchmark_data_smithi031_467097_object133:head# 2024-04-25T09:19:44.251 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:4fc94315:::benchmark_data_smithi031_467097_object82:head# 2024-04-25T09:19:44.252 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:503f3b23:::benchmark_data_smithi031_467097_object407:head# 2024-04-25T09:19:44.252 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:50fcb57f:::benchmark_data_smithi031_467097_object329:head# 2024-04-25T09:19:44.253 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51b06ca8:::benchmark_data_smithi031_467097_object146:head# 2024-04-25T09:19:44.253 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51bcde7e:::benchmark_data_smithi031_467097_object110:head# 2024-04-25T09:19:44.254 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:51e5bee3:::benchmark_data_smithi031_467097_object349:head# 2024-04-25T09:19:44.255 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:53259d4e:::benchmark_data_smithi031_467097_object276:head# 2024-04-25T09:19:44.255 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5383104a:::benchmark_data_smithi031_467097_object240:head# 2024-04-25T09:19:44.255 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:54236b4b:::benchmark_data_smithi031_467097_object227:head# 2024-04-25T09:19:44.256 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5457fc8f:::benchmark_data_smithi031_467097_object182:head# 2024-04-25T09:19:44.256 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:548176e3:::benchmark_data_smithi031_467097_object53:head# 2024-04-25T09:19:44.257 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:54e65407:::benchmark_data_smithi031_467097_object360:head# 2024-04-25T09:19:44.258 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:554edb45:::benchmark_data_smithi031_467097_object88:head# 2024-04-25T09:19:44.259 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:55565cdc:::benchmark_data_smithi031_467097_object363:head# 2024-04-25T09:19:44.259 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:55a8782c:::benchmark_data_smithi031_467097_object226:head# 2024-04-25T09:19:44.260 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:55b52fde:::benchmark_data_smithi031_467097_object264:head# 2024-04-25T09:19:44.260 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56096565:::benchmark_data_smithi031_467097_object304:head# 2024-04-25T09:19:44.261 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5636a188:::benchmark_data_smithi031_467097_object203:head# 2024-04-25T09:19:44.261 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5678a988:::benchmark_data_smithi031_467097_object136:head# 2024-04-25T09:19:44.262 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:56d8028f:::benchmark_data_smithi031_467097_object131:head# 2024-04-25T09:19:44.263 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57ab9ba6:::benchmark_data_smithi031_467097_object76:head# 2024-04-25T09:19:44.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57cb25cd:::benchmark_data_smithi031_467097_object178:head# 2024-04-25T09:19:44.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:57f7ab1b:::benchmark_data_smithi031_467097_object160:head# 2024-04-25T09:19:44.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:592e58e9:::benchmark_data_smithi031_467097_object91:head# 2024-04-25T09:19:44.265 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:59ca84b1:::benchmark_data_smithi031_467097_object31:head# 2024-04-25T09:19:44.266 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:59e8d3ef:::benchmark_data_smithi031_467097_object120:head# 2024-04-25T09:19:44.266 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5a086d35:::benchmark_data_smithi031_467097_object60:head# 2024-04-25T09:19:44.267 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5ac0437b:::benchmark_data_smithi031_467097_object291:head# 2024-04-25T09:19:44.268 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5b112f35:::benchmark_data_smithi031_467097_object69:head# 2024-04-25T09:19:44.268 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5b6c0411:::benchmark_data_smithi031_467097_object167:head# 2024-04-25T09:19:44.268 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5c372236:::benchmark_data_smithi031_467097_object279:head# 2024-04-25T09:19:44.269 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e0390c8:::benchmark_data_smithi031_467097_object281:head# 2024-04-25T09:19:44.269 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:5e6ccbf9:::benchmark_data_smithi031_467097_object238:head# 2024-04-25T09:19:44.270 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:602f4300:::benchmark_data_smithi031_467097_object344:head# 2024-04-25T09:19:44.271 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:60910817:::benchmark_data_smithi031_467097_object159:head# 2024-04-25T09:19:44.271 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:635636bd:::benchmark_data_smithi031_467097_object15:head# 2024-04-25T09:19:44.272 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6433df6e:::benchmark_data_smithi031_467097_object337:head# 2024-04-25T09:19:44.272 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:644bc9ac:::benchmark_data_smithi031_467097_object127:head# 2024-04-25T09:19:44.273 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:648892a5:::benchmark_data_smithi031_467097_object408:head# 2024-04-25T09:19:44.273 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:65117d3d:::benchmark_data_smithi031_467097_object401:head# 2024-04-25T09:19:44.275 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6546cc3d:::benchmark_data_smithi031_467097_object366:head# 2024-04-25T09:19:44.276 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:66a12924:::benchmark_data_smithi031_467097_object197:head# 2024-04-25T09:19:44.276 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:66fcb472:::benchmark_data_smithi031_467097_object157:head# 2024-04-25T09:19:44.276 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6748c1cb:::benchmark_data_smithi031_467097_object403:head# 2024-04-25T09:19:44.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6755413c:::benchmark_data_smithi031_467097_object43:head# 2024-04-25T09:19:44.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:679c2cbe:::benchmark_data_smithi031_467097_object114:head# 2024-04-25T09:19:44.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:67d213ea:::benchmark_data_smithi031_467097_object296:head# 2024-04-25T09:19:44.278 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:67d43a1b:::benchmark_data_smithi031_467097_object164:head# 2024-04-25T09:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:698548d2:::benchmark_data_smithi031_467097_object380:head# 2024-04-25T09:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:69c0acfb:::benchmark_data_smithi031_467097_object364:head# 2024-04-25T09:19:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6b6a7d9d:::benchmark_data_smithi031_467097_object413:head# 2024-04-25T09:19:44.280 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6b7e4d65:::benchmark_data_smithi031_467097_object300:head# 2024-04-25T09:19:44.280 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6b95753d:::benchmark_data_smithi031_467097_object140:head# 2024-04-25T09:19:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ba8f48f:::benchmark_data_smithi031_467097_object135:head# 2024-04-25T09:19:44.281 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6c22f730:::benchmark_data_smithi031_467097_object368:head# 2024-04-25T09:19:44.282 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ca30cb8:::benchmark_data_smithi031_467097_object34:head# 2024-04-25T09:19:44.282 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6ce7d6cf:::benchmark_data_smithi031_467097_object395:head# 2024-04-25T09:19:44.283 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6d8590c1:::benchmark_data_smithi031_467097_object98:head# 2024-04-25T09:19:44.283 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6e9737ec:::benchmark_data_smithi031_467097_object246:head# 2024-04-25T09:19:44.284 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:6f70fa66:::benchmark_data_smithi031_467097_object212:head# 2024-04-25T09:19:44.284 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7097b6e5:::benchmark_data_smithi031_467097_object301:head# 2024-04-25T09:19:44.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7123e38f:::benchmark_data_smithi031_467097_object225:head# 2024-04-25T09:19:44.285 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7236a67c:::benchmark_data_smithi031_467097_object162:head# 2024-04-25T09:19:44.286 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72816d07:::benchmark_data_smithi031_467097_object190:head# 2024-04-25T09:19:44.287 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:72b5fb7f:::benchmark_data_smithi031_467097_object155:head# 2024-04-25T09:19:44.288 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7394813d:::benchmark_data_smithi031_467097_object321:head# 2024-04-25T09:19:44.288 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:73ec1cbd:::benchmark_data_smithi031_467097_object325:head# 2024-04-25T09:19:44.289 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:75135d5c:::benchmark_data_smithi031_467097_object351:head# 2024-04-25T09:19:44.290 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:752ed385:::benchmark_data_smithi031_467097_object254:head# 2024-04-25T09:19:44.290 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:755b566b:::benchmark_data_smithi031_467097_object196:head# 2024-04-25T09:19:44.291 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7583b304:::benchmark_data_smithi031_467097_object315:head# 2024-04-25T09:19:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:76601c60:::benchmark_data_smithi031_467097_object273:head# 2024-04-25T09:19:44.292 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:774dc987:::benchmark_data_smithi031_467097_object402:head# 2024-04-25T09:19:44.293 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:78883ce8:::benchmark_data_smithi031_467097_object7:head# 2024-04-25T09:19:44.294 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:79f84ae2:::benchmark_data_smithi031_467097_object80:head# 2024-04-25T09:19:44.294 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7a2864bd:::benchmark_data_smithi031_467097_object144:head# 2024-04-25T09:19:44.295 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7a55de80:::benchmark_data_smithi031_467097_object341:head# 2024-04-25T09:19:44.296 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7a63eab7:::benchmark_data_smithi031_467097_object255:head# 2024-04-25T09:19:44.296 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7aa2a2b9:::benchmark_data_smithi031_467097_object220:head# 2024-04-25T09:19:44.296 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7bdc4396:::benchmark_data_smithi031_467097_object11:head# 2024-04-25T09:19:44.298 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7c05decd:::benchmark_data_smithi031_467097_object106:head# 2024-04-25T09:19:44.298 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7cd0072f:::benchmark_data_smithi031_467097_object3:head# 2024-04-25T09:19:44.298 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7ce387ef:::benchmark_data_smithi031_467097_object26:head# 2024-04-25T09:19:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dd07c2e:::benchmark_data_smithi031_467097_object243:head# 2024-04-25T09:19:44.299 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7dda6d00:::benchmark_data_smithi031_467097_object340:head# 2024-04-25T09:19:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7e06874c:::benchmark_data_smithi031_467097_object405:head# 2024-04-25T09:19:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7e4a1e32:::benchmark_data_smithi031_467097_object388:head# 2024-04-25T09:19:44.301 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7e6d8241:::benchmark_data_smithi031_467097_object209:head# 2024-04-25T09:19:44.302 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7e733df6:::benchmark_data_smithi031_467097_object45:head# 2024-04-25T09:19:44.303 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:7f9dad66:::benchmark_data_smithi031_467097_object221:head# 2024-04-25T09:19:44.303 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:80b7356a:::benchmark_data_smithi031_467097_object90:head# 2024-04-25T09:19:44.303 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:82355ecb:::benchmark_data_smithi031_467097_object261:head# 2024-04-25T09:19:44.304 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:826dcb52:::benchmark_data_smithi031_467097_object381:head# 2024-04-25T09:19:44.306 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:832bd708:::benchmark_data_smithi031_467097_object41:head# 2024-04-25T09:19:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:834ab85e:::benchmark_data_smithi031_467097_object260:head# 2024-04-25T09:19:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8429b72a:::benchmark_data_smithi031_467097_object263:head# 2024-04-25T09:19:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:85406674:::benchmark_data_smithi031_467097_object125:head# 2024-04-25T09:19:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8764e146:::benchmark_data_smithi031_467097_object107:head# 2024-04-25T09:19:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8793b74f:::benchmark_data_smithi031_467097_object50:head# 2024-04-25T09:19:44.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:87947e07:::benchmark_data_smithi031_467097_object365:head# 2024-04-25T09:19:44.311 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:87be37bd:::benchmark_data_smithi031_467097_object100:head# 2024-04-25T09:19:44.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:87e14a46:::benchmark_data_smithi031_467097_object232:head# 2024-04-25T09:19:44.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8897234b:::benchmark_data_smithi031_467097_object5:head# 2024-04-25T09:19:44.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:88f7b731:::benchmark_data_smithi031_467097_object68:head# 2024-04-25T09:19:44.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:895edafe:::benchmark_data_smithi031_467097_object256:head# 2024-04-25T09:19:44.312 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:89616ef6:::benchmark_data_smithi031_467097_object251:head# 2024-04-25T09:19:44.313 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8a15ee8a:::benchmark_data_smithi031_467097_object79:head# 2024-04-25T09:19:44.313 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8a1c6edb:::benchmark_data_smithi031_467097_object54:head# 2024-04-25T09:19:44.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8bd251ab:::benchmark_data_smithi031_467097_object77:head# 2024-04-25T09:19:44.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8c2b6839:::benchmark_data_smithi031_467097_object115:head# 2024-04-25T09:19:44.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8c3c1f2a:::benchmark_data_smithi031_467097_object338:head# 2024-04-25T09:19:44.321 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8c9bc6bb:::benchmark_data_smithi031_467097_object130:head# 2024-04-25T09:19:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8ca24803:::benchmark_data_smithi031_467097_object13:head# 2024-04-25T09:19:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8d8b1ec0:::benchmark_data_smithi031_467097_object161:head# 2024-04-25T09:19:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8e26ca72:::benchmark_data_smithi031_467097_object58:head# 2024-04-25T09:19:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8e68503c:::benchmark_data_smithi031_467097_object350:head# 2024-04-25T09:19:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:8eb87e8b:::benchmark_data_smithi031_467097_object347:head# 2024-04-25T09:19:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9086a3ff:::benchmark_data_smithi031_467097_object61:head# 2024-04-25T09:19:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:91cdb8e7:::benchmark_data_smithi031_467097_object372:head# 2024-04-25T09:19:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:92b10e83:::benchmark_data_smithi031_467097_object67:head# 2024-04-25T09:19:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:92de843c:::benchmark_data_smithi031_467097_object354:head# 2024-04-25T09:19:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:93050ebf:::benchmark_data_smithi031_467097_object309:head# 2024-04-25T09:19:44.323 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9428fa05:::benchmark_data_smithi031_467097_object250:head# 2024-04-25T09:19:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:94839f49:::benchmark_data_smithi031_467097_object358:head# 2024-04-25T09:19:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:961116de:::benchmark_data_smithi031_467097_object379:head# 2024-04-25T09:19:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:96c6c36c:::benchmark_data_smithi031_467097_object200:head# 2024-04-25T09:19:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:973d4c08:::benchmark_data_smithi031_467097_object216:head# 2024-04-25T09:19:44.324 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:97544e42:::benchmark_data_smithi031_467097_object32:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9812b2a7:::benchmark_data_smithi031_467097_object373:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:983c9d6b:::benchmark_data_smithi031_467097_object84:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9848f2b2:::benchmark_data_smithi031_467097_object27:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9ade755f:::benchmark_data_smithi031_467097_object312:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9b69017a:::benchmark_data_smithi031_467097_object327:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9bc5a763:::benchmark_data_smithi031_467097_object174:head# 2024-04-25T09:19:44.332 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9c08b79c:::benchmark_data_smithi031_467097_object362:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9d92b034:::benchmark_data_smithi031_467097_object92:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9deff396:::benchmark_data_smithi031_467097_object20:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9e06d86e:::benchmark_data_smithi031_467097_object409:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:9f474749:::benchmark_data_smithi031_467097_object284:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a20531ba:::benchmark_data_smithi031_467097_object176:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a2798eba:::benchmark_data_smithi031_467097_object334:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a3370137:::benchmark_data_smithi031_467097_object118:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a38ab8b7:::benchmark_data_smithi031_467097_object81:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a397cde8:::benchmark_data_smithi031_467097_object285:head# 2024-04-25T09:19:44.333 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a437abdd:::benchmark_data_smithi031_467097_object416:head# 2024-04-25T09:19:44.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a443327d:::benchmark_data_smithi031_467097_object320:head# 2024-04-25T09:19:44.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a500614c:::benchmark_data_smithi031_467097_object229:head# 2024-04-25T09:19:44.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a55c0d56:::benchmark_data_smithi031_467097_object414:head# 2024-04-25T09:19:44.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a6257be7:::benchmark_data_smithi031_467097_object346:head# 2024-04-25T09:19:44.334 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a8bff898:::benchmark_data_smithi031_467097_object215:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a8e9e624:::benchmark_data_smithi031_467097_object87:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a96a943e:::benchmark_data_smithi031_467097_object262:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a9a387bd:::benchmark_data_smithi031_467097_object324:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:a9fa01d4:::benchmark_data_smithi031_467097_object51:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aa071e7e:::benchmark_data_smithi031_467097_object258:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aa1da460:::benchmark_data_smithi031_467097_object399:head# 2024-04-25T09:19:44.336 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ab1b18f5:::benchmark_data_smithi031_467097_object46:head# 2024-04-25T09:19:44.338 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:abb2758c:::benchmark_data_smithi031_467097_object10:head# 2024-04-25T09:19:44.338 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:acfb2156:::benchmark_data_smithi031_467097_object339:head# 2024-04-25T09:19:44.338 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aebe48ef:::benchmark_data_smithi031_467097_object95:head# 2024-04-25T09:19:44.338 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:aed45aba:::benchmark_data_smithi031_467097_object330:head# 2024-04-25T09:19:44.339 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:af4f7dcb:::benchmark_data_smithi031_467097_object170:head# 2024-04-25T09:19:44.339 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b08ffde3:::benchmark_data_smithi031_467097_object24:head# 2024-04-25T09:19:44.340 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b0bb418b:::benchmark_data_smithi031_467097_object323:head# 2024-04-25T09:19:44.340 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b1859ded:::benchmark_data_smithi031_467097_object28:head# 2024-04-25T09:19:44.341 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b1fa9eb4:::benchmark_data_smithi031_467097_object64:head# 2024-04-25T09:19:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b3dd1b29:::benchmark_data_smithi031_467097_object332:head# 2024-04-25T09:19:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b3ee5179:::benchmark_data_smithi031_467097_object111:head# 2024-04-25T09:19:44.342 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b46e37e5:::benchmark_data_smithi031_467097_object177:head# 2024-04-25T09:19:44.343 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b4788b5b:::benchmark_data_smithi031_467097_object208:head# 2024-04-25T09:19:44.343 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b5dea728:::benchmark_data_smithi031_467097_object201:head# 2024-04-25T09:19:44.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b62e4598:::benchmark_data_smithi031_467097_object267:head# 2024-04-25T09:19:44.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b70d191c:::benchmark_data_smithi031_467097_object292:head# 2024-04-25T09:19:44.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b83247a1:::benchmark_data_smithi031_467097_object286:head# 2024-04-25T09:19:44.344 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:19:44.345 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:b9f066f9:::benchmark_data_smithi031_467097_object343:head# 2024-04-25T09:19:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ba876785:::benchmark_data_smithi031_467097_object252:head# 2024-04-25T09:19:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bac5ac8b:::benchmark_data_smithi031_467097_object112:head# 2024-04-25T09:19:44.346 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bafd965c:::benchmark_data_smithi031_467097_object96:head# 2024-04-25T09:19:44.347 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb11d564:::benchmark_data_smithi031_467097_object288:head# 2024-04-25T09:19:44.347 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb672e07:::benchmark_data_smithi031_467097_object361:head# 2024-04-25T09:19:44.348 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bb6db15c:::benchmark_data_smithi031_467097_object355:head# 2024-04-25T09:19:44.349 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bc34dff4:::benchmark_data_smithi031_467097_object185:head# 2024-04-25T09:19:44.349 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bc931565:::benchmark_data_smithi031_467097_object158:head# 2024-04-25T09:19:44.350 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bd41ee77:::benchmark_data_smithi031_467097_object266:head# 2024-04-25T09:19:44.350 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:be83025a:::benchmark_data_smithi031_467097_object397:head# 2024-04-25T09:19:44.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:be915db4:::benchmark_data_smithi031_467097_object290:head# 2024-04-25T09:19:44.351 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bea2df34:::benchmark_data_smithi031_467097_object121:head# 2024-04-25T09:19:44.353 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bf1f6802:::benchmark_data_smithi031_467097_object219:head# 2024-04-25T09:19:44.353 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:bf5dd63e:::benchmark_data_smithi031_467097_object109:head# 2024-04-25T09:19:44.354 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c0fb5b88:::benchmark_data_smithi031_467097_object205:head# 2024-04-25T09:19:44.355 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c14cc90b:::benchmark_data_smithi031_467097_object94:head# 2024-04-25T09:19:44.356 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c157d7c3:::benchmark_data_smithi031_467097_object19:head# 2024-04-25T09:19:44.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c1bf13c4:::benchmark_data_smithi031_467097_object37:head# 2024-04-25T09:19:44.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c1d86d32:::benchmark_data_smithi031_467097_object269:head# 2024-04-25T09:19:44.357 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c1e3f260:::benchmark_data_smithi031_467097_object275:head# 2024-04-25T09:19:44.358 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c27f1e0a:::benchmark_data_smithi031_467097_object375:head# 2024-04-25T09:19:44.358 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c2ce63b1:::benchmark_data_smithi031_467097_object326:head# 2024-04-25T09:19:44.359 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c3c19702:::benchmark_data_smithi031_467097_object147:head# 2024-04-25T09:19:44.359 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c3f54765:::benchmark_data_smithi031_467097_object48:head# 2024-04-25T09:19:44.360 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c4049b17:::benchmark_data_smithi031_467097_object9:head# 2024-04-25T09:19:44.360 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c4546446:::benchmark_data_smithi031_467097_object234:head# 2024-04-25T09:19:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c46b9f43:::benchmark_data_smithi031_467097_object75:head# 2024-04-25T09:19:44.361 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c52ac604:::benchmark_data_smithi031_467097_object199:head# 2024-04-25T09:19:44.362 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c603b7d6:::benchmark_data_smithi031_467097_object198:head# 2024-04-25T09:19:44.363 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c63fe32b:::benchmark_data_smithi031_467097_object244:head# 2024-04-25T09:19:44.363 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c7da009d:::benchmark_data_smithi031_467097_object23:head# 2024-04-25T09:19:44.364 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c8d1401f:::benchmark_data_smithi031_467097_object124:head# 2024-04-25T09:19:44.364 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c9ee5ab7:::benchmark_data_smithi031_467097_object253:head# 2024-04-25T09:19:44.365 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:c9ff2023:::benchmark_data_smithi031_467097_object318:head# 2024-04-25T09:19:44.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca11222e:::benchmark_data_smithi031_467097_object245:head# 2024-04-25T09:19:44.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca467a5a:::benchmark_data_smithi031_467097_object298:head# 2024-04-25T09:19:44.367 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca504494:::benchmark_data_smithi031_467097_object328:head# 2024-04-25T09:19:44.368 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca518a1d:::benchmark_data_smithi031_467097_object108:head# 2024-04-25T09:19:44.368 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ca76242d:::benchmark_data_smithi031_467097_object202:head# 2024-04-25T09:19:44.369 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cabf5cf4:::benchmark_data_smithi031_467097_object308:head# 2024-04-25T09:19:44.370 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cb59f22b:::benchmark_data_smithi031_467097_object30:head# 2024-04-25T09:19:44.370 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cb72c784:::benchmark_data_smithi031_467097_object377:head# 2024-04-25T09:19:44.370 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cc2e932a:::benchmark_data_smithi031_467097_object265:head# 2024-04-25T09:19:44.371 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cd0c4b72:::benchmark_data_smithi031_467097_object376:head# 2024-04-25T09:19:44.372 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ce6786b9:::benchmark_data_smithi031_467097_object44:head# 2024-04-25T09:19:44.372 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ced3112b:::benchmark_data_smithi031_467097_object242:head# 2024-04-25T09:19:44.373 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cf735205:::benchmark_data_smithi031_467097_object168:head# 2024-04-25T09:19:44.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:cf81d81f:::benchmark_data_smithi031_467097_object1:head# 2024-04-25T09:19:44.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d0c5feda:::benchmark_data_smithi031_467097_object210:head# 2024-04-25T09:19:44.374 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d0cea92e:::benchmark_data_smithi031_467097_object303:head# 2024-04-25T09:19:44.375 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d14da95c:::benchmark_data_smithi031_467097_object307:head# 2024-04-25T09:19:44.375 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d165a956:::benchmark_data_smithi031_467097_object89:head# 2024-04-25T09:19:44.376 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d17ee123:::benchmark_data_smithi031_467097_object287:head# 2024-04-25T09:19:44.376 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d1d9b532:::benchmark_data_smithi031_467097_object57:head# 2024-04-25T09:19:44.377 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d1e2873f:::benchmark_data_smithi031_467097_object129:head# 2024-04-25T09:19:44.377 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d2af251c:::benchmark_data_smithi031_467097_object294:head# 2024-04-25T09:19:44.378 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d3d8c1e5:::benchmark_data_smithi031_467097_object228:head# 2024-04-25T09:19:44.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d47becaf:::benchmark_data_smithi031_467097_object398:head# 2024-04-25T09:19:44.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d4d48e4c:::benchmark_data_smithi031_467097_object352:head# 2024-04-25T09:19:44.379 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d4f0a3d2:::benchmark_data_smithi031_467097_object384:head# 2024-04-25T09:19:44.380 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d4fabdb8:::benchmark_data_smithi031_467097_object314:head# 2024-04-25T09:19:44.380 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d5252a4b:::benchmark_data_smithi031_467097_object322:head# 2024-04-25T09:19:44.381 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d6826dbd:::benchmark_data_smithi031_467097_object104:head# 2024-04-25T09:19:44.382 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:d9d97115:::benchmark_data_smithi031_467097_object222:head# 2024-04-25T09:19:44.383 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:da9a2295:::benchmark_data_smithi031_467097_object116:head# 2024-04-25T09:19:44.383 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:daa1228a:::benchmark_data_smithi031_467097_object374:head# 2024-04-25T09:19:44.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dabcbbca:::benchmark_data_smithi031_467097_object25:head# 2024-04-25T09:19:44.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dcbce9ae:::benchmark_data_smithi031_467097_object302:head# 2024-04-25T09:19:44.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dcebadbc:::benchmark_data_smithi031_467097_object86:head# 2024-04-25T09:19:44.384 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd8b0387:::benchmark_data_smithi031_467097_object194:head# 2024-04-25T09:19:44.385 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:dd8c5adf:::benchmark_data_smithi031_467097_object163:head# 2024-04-25T09:19:44.385 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ddb4c7d1:::benchmark_data_smithi031_467097_object378:head# 2024-04-25T09:19:44.386 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:de11411f:::benchmark_data_smithi031_467097_object99:head# 2024-04-25T09:19:44.387 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:deccc71b:::benchmark_data_smithi031_467097_object195:head# 2024-04-25T09:19:44.387 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ded9d547:::benchmark_data_smithi031_467097_object357:head# 2024-04-25T09:19:44.387 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:df4c3bad:::benchmark_data_smithi031_467097_object132:head# 2024-04-25T09:19:44.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:df94ff48:::benchmark_data_smithi031_467097_object52:head# 2024-04-25T09:19:44.388 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e01aac02:::benchmark_data_smithi031_467097_object217:head# 2024-04-25T09:19:44.389 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e023c0bf:::benchmark_data_smithi031_467097_object2:head# 2024-04-25T09:19:44.389 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e03c6ba0:::benchmark_data_smithi031_467097_object387:head# 2024-04-25T09:19:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e0404815:::benchmark_data_smithi031_467097_object6:head# 2024-04-25T09:19:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e158c29b:::benchmark_data_smithi031_467097_object35:head# 2024-04-25T09:19:44.390 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e1ade39b:::benchmark_data_smithi031_467097_object21:head# 2024-04-25T09:19:44.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e2c51ff4:::benchmark_data_smithi031_467097_object85:head# 2024-04-25T09:19:44.391 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e2cafd8f:::benchmark_data_smithi031_467097_object223:head# 2024-04-25T09:19:44.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e314ee7c:::benchmark_data_smithi031_467097_object74:head# 2024-04-25T09:19:44.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e39ec5ea:::benchmark_data_smithi031_467097_object83:head# 2024-04-25T09:19:44.392 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e48acdbc:::benchmark_data_smithi031_467097_object65:head# 2024-04-25T09:19:44.393 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e49798ea:::benchmark_data_smithi031_467097_object207:head# 2024-04-25T09:19:44.393 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e677fc79:::benchmark_data_smithi031_467097_object173:head# 2024-04-25T09:19:44.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e737ca82:::benchmark_data_smithi031_467097_object47:head# 2024-04-25T09:19:44.394 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e7522751:::benchmark_data_smithi031_467097_object186:head# 2024-04-25T09:19:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e7617382:::benchmark_data_smithi031_467097_object404:head# 2024-04-25T09:19:44.395 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e85963b8:::benchmark_data_smithi031_467097_object310:head# 2024-04-25T09:19:44.396 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:e8924085:::benchmark_data_smithi031_467097_object17:head# 2024-04-25T09:19:44.396 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:eac421ca:::benchmark_data_smithi031_467097_object14:head# 2024-04-25T09:19:44.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:eb26f6d7:::benchmark_data_smithi031_467097_object356:head# 2024-04-25T09:19:44.397 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ee0210d1:::benchmark_data_smithi031_467097_object70:head# 2024-04-25T09:19:44.398 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ee410e68:::benchmark_data_smithi031_467097_object180:head# 2024-04-25T09:19:44.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ef2b8b1f:::benchmark_data_smithi031_467097_object101:head# 2024-04-25T09:19:44.399 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f26dffcd:::benchmark_data_smithi031_467097_object204:head# 2024-04-25T09:19:44.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f4c7ad45:::benchmark_data_smithi031_467097_object66:head# 2024-04-25T09:19:44.400 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f4e38a5e:::benchmark_data_smithi031_467097_object39:head# 2024-04-25T09:19:44.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f5ae7d63:::benchmark_data_smithi031_467097_object274:head# 2024-04-25T09:19:44.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f5cb5ac6:::benchmark_data_smithi031_467097_object78:head# 2024-04-25T09:19:44.401 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f62e2160:::benchmark_data_smithi031_467097_object152:head# 2024-04-25T09:19:44.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f655c7e6:::benchmark_data_smithi031_467097_object141:head# 2024-04-25T09:19:44.402 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f65a695f:::benchmark_data_smithi031_467097_object63:head# 2024-04-25T09:19:44.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f6fdedb6:::benchmark_data_smithi031_467097_object122:head# 2024-04-25T09:19:44.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f7516ecf:::benchmark_data_smithi031_467097_object391:head# 2024-04-25T09:19:44.403 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f7d922b0:::benchmark_data_smithi031_467097_object259:head# 2024-04-25T09:19:44.404 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f7e08e1e:::benchmark_data_smithi031_467097_object386:head# 2024-04-25T09:19:44.404 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f7e6d340:::benchmark_data_smithi031_467097_object0:head# 2024-04-25T09:19:44.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f8530af6:::benchmark_data_smithi031_467097_object97:head# 2024-04-25T09:19:44.405 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f86e4a3f:::benchmark_data_smithi031_467097_object179:head# 2024-04-25T09:19:44.406 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f96e7e7a:::benchmark_data_smithi031_467097_object331:head# 2024-04-25T09:19:44.407 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:f9d769e5:::benchmark_data_smithi031_467097_object412:head# 2024-04-25T09:19:44.407 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fa441986:::benchmark_data_smithi031_467097_object8:head# 2024-04-25T09:19:44.408 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:faa9086e:::benchmark_data_smithi031_467097_object93:head# 2024-04-25T09:19:44.408 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:faab5dd1:::benchmark_data_smithi031_467097_object102:head# 2024-04-25T09:19:44.408 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fac1e97a:::benchmark_data_smithi031_467097_object335:head# 2024-04-25T09:19:44.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb0c180e:::benchmark_data_smithi031_467097_object126:head# 2024-04-25T09:19:44.409 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fb9f7ad8:::benchmark_data_smithi031_467097_object280:head# 2024-04-25T09:19:44.410 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fce0f0a2:::benchmark_data_smithi031_467097_object247:head# 2024-04-25T09:19:44.410 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fddbf60a:::benchmark_data_smithi031_467097_object371:head# 2024-04-25T09:19:44.411 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fdfa9640:::benchmark_data_smithi031_467097_object316:head# 2024-04-25T09:19:44.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:fe7e28e5:::benchmark_data_smithi031_467097_object392:head# 2024-04-25T09:19:44.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:feb36f49:::benchmark_data_smithi031_467097_object282:head# 2024-04-25T09:19:44.412 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:feb403bf:::benchmark_data_smithi031_467097_object151:head# 2024-04-25T09:19:44.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ff3a0c02:::benchmark_data_smithi031_467097_object189:head# 2024-04-25T09:19:44.413 INFO:tasks.workunit.client.0.smithi031.stderr:Read #1:ffbbf3bd:::benchmark_data_smithi031_467097_object406:head# 2024-04-25T09:19:44.414 INFO:tasks.workunit.client.0.smithi031.stderr:Export successful 2024-04-25T09:19:44.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:19:44.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:19:44.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:19:44.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:19:44.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:19:44.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:44.710 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:44.711 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:44.711 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:19:44.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:19:44.713 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:19:44.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:19:44.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:19:44.714 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:19:44.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:19:44.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:19:44.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:19:44.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:19:44.749 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:44.761+0000 7f14c34137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:44.752 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:44.765+0000 7f14c34137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:44.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:44.765+0000 7f14c34137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:19:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:19:45.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:19:45.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:19:45.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:19:45.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:45.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:19:45.189 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:19:45.189 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:45.190 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:45.299 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:45.309+0000 7f14c34137c0 -1 Falling back to public interface 2024-04-25T09:19:45.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:46.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:46.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:46.628 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:19:46.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:46.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:46.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:47.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:48.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:48.075 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:19:48.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:48.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:48.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:48.204 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:19:48.217+0000 7f14c34137c0 -1 osd.0 9 log_to_monitors true 2024-04-25T09:19:48.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:49.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:49.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:49.549 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:19:49.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:49.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:49.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:50.015 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/851896419,v1:127.0.0.1:6803/851896419] [v2:127.0.0.1:6804/851896419,v1:127.0.0.1:6805/851896419] exists,up 7e17a2f7-7821-4592-a394-a058baf89f2d 2024-04-25T09:19:50.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:19:50.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:19:50.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:19:50.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-04-25T09:19:50.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-04-25T09:19:55.823 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-04-25T09:19:55.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-04-25T09:19:55.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-04-25T09:19:55.968 INFO:tasks.workunit.client.0.smithi031.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:19:55.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-04-25T09:19:55.970 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:19:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:19:55.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:19:55.971 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:19:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:19:55.972 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:19:55.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:19:55.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:19:55.972 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:19:56.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:19:56.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:19:56.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:19:56.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:19:56.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:19:56.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:19:56.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:19:56.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:19:56.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:19:56.526 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:19:56.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607556 2024-04-25T09:19:56.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607556 2024-04-25T09:19:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-51539607556' 2024-04-25T09:19:56.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:19:56.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-51539607556 2024-04-25T09:19:56.689 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:19:56.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:19:56.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-51539607556 2024-04-25T09:19:56.691 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:19:56.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607556 2024-04-25T09:19:56.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-04-25T09:19:56.692 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 51539607556 2024-04-25T09:19:56.692 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:19:57.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607554 -lt 51539607556 2024-04-25T09:19:57.129 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:19:58.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:19:58.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:19:58.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607554 -lt 51539607556 2024-04-25T09:19:58.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:19:59.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-25T09:19:59.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:00.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607557 -lt 51539607556 2024-04-25T09:20:00.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:20:00.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:00.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:20:00.709 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:20:01.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:20:01.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:20:01.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:01.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:01.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:20:01.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:20:01.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:20:01.973 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:20:03.145 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:20:03.145 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:20:03.146 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:20:03.470 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T09:20:03.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:20:04.911 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:20:04.911 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T09:20:04.912 INFO:tasks.workunit.client.0.smithi031.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:20:04.912 INFO:tasks.workunit.client.0.smithi031.stderr: children are 1.1 2024-04-25T09:20:04.912 INFO:tasks.workunit.client.0.smithi031.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-04-25T09:20:05.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:20:06.052 INFO:tasks.workunit.client.0.smithi031.stdout: marking collection for removal 2024-04-25T09:20:06.053 INFO:tasks.workunit.client.0.smithi031.stdout:setting '_remove' omap key 2024-04-25T09:20:06.053 INFO:tasks.workunit.client.0.smithi031.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:20:06.385 INFO:tasks.workunit.client.0.smithi031.stdout:Remove successful 2024-04-25T09:20:06.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:20:07.623 INFO:tasks.workunit.client.0.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:20:07.623 INFO:tasks.workunit.client.0.smithi031.stdout:Importing pgid 1.0 2024-04-25T09:20:07.624 INFO:tasks.workunit.client.0.smithi031.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:20:07.625 INFO:tasks.workunit.client.0.smithi031.stderr: children are 1.1 2024-04-25T09:20:07.625 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:006fbae5:::benchmark_data_smithi031_467097_object305:head# 2024-04-25T09:20:07.625 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.687 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00dc8f94:::benchmark_data_smithi031_467097_object231:head# 2024-04-25T09:20:07.688 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.721 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:00e4b61e:::benchmark_data_smithi031_467097_object192:head# 2024-04-25T09:20:07.721 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.754 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:01853150:::benchmark_data_smithi031_467097_object123:head# 2024-04-25T09:20:07.755 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.788 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:02029128:::benchmark_data_smithi031_467097_object33:head# 2024-04-25T09:20:07.788 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.821 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:035e3271:::benchmark_data_smithi031_467097_object336:head# 2024-04-25T09:20:07.821 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.855 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:042d88ac:::benchmark_data_smithi031_467097_object175:head# 2024-04-25T09:20:07.855 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.888 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:050ee465:::benchmark_data_smithi031_467097_object71:head# 2024-04-25T09:20:07.888 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.922 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:05563abd:::benchmark_data_smithi031_467097_object193:head# 2024-04-25T09:20:07.922 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.955 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0650a180:::benchmark_data_smithi031_467097_object345:head# 2024-04-25T09:20:07.955 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:07.988 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:06e12f21:::benchmark_data_smithi031_467097_object128:head# 2024-04-25T09:20:07.989 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.022 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0735b904:::benchmark_data_smithi031_467097_object311:head# 2024-04-25T09:20:08.022 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.055 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:07baf8c0:::benchmark_data_smithi031_467097_object165:head# 2024-04-25T09:20:08.055 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.089 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:07cbb58b:::benchmark_data_smithi031_467097_object249:head# 2024-04-25T09:20:08.089 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.122 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:08aa4630:::benchmark_data_smithi031_467097_object257:head# 2024-04-25T09:20:08.122 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.156 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:092a861b:::benchmark_data_smithi031_467097_object191:head# 2024-04-25T09:20:08.156 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.198 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:092fb541:::benchmark_data_smithi031_467097_object139:head# 2024-04-25T09:20:08.198 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.231 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:09c6b1c6:::benchmark_data_smithi031_467097_object359:head# 2024-04-25T09:20:08.231 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.264 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0b4938ac:::benchmark_data_smithi031_467097_object171:head# 2024-04-25T09:20:08.265 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.298 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ba7118e:::benchmark_data_smithi031_467097_object59:head# 2024-04-25T09:20:08.298 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.340 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0cc5814b:::benchmark_data_smithi031_467097_object153:head# 2024-04-25T09:20:08.340 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.365 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ccd0c3e:::benchmark_data_smithi031_467097_object389:head# 2024-04-25T09:20:08.365 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.390 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0d1840e3:::benchmark_data_smithi031_467097_object272:head# 2024-04-25T09:20:08.390 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.423 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0ded6035:::benchmark_data_smithi031_467097_object393:head# 2024-04-25T09:20:08.423 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.457 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0e6631d3:::benchmark_data_smithi031_467097_object400:head# 2024-04-25T09:20:08.457 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.490 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:0e9d9805:::benchmark_data_smithi031_467097_object18:head# 2024-04-25T09:20:08.491 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.524 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1088b06d:::benchmark_data_smithi031_467097_object239:head# 2024-04-25T09:20:08.524 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.557 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:123cbe44:::benchmark_data_smithi031_467097_object12:head# 2024-04-25T09:20:08.557 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.591 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:124e586f:::benchmark_data_smithi031_467097_object313:head# 2024-04-25T09:20:08.591 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.624 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:12c4d666:::benchmark_data_smithi031_467097_object214:head# 2024-04-25T09:20:08.624 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.674 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:12f7f36c:::benchmark_data_smithi031_467097_object236:head# 2024-04-25T09:20:08.675 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.724 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:13ac91cc:::benchmark_data_smithi031_467097_object353:head# 2024-04-25T09:20:08.725 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.750 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:14dc840c:::benchmark_data_smithi031_467097_object181:head# 2024-04-25T09:20:08.750 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.775 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1572997c:::benchmark_data_smithi031_467097_object166:head# 2024-04-25T09:20:08.775 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.817 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:158037c9:::benchmark_data_smithi031_467097_object333:head# 2024-04-25T09:20:08.817 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.842 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1604cc8a:::benchmark_data_smithi031_467097_object370:head# 2024-04-25T09:20:08.842 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1638a65d:::benchmark_data_smithi031_467097_object183:head# 2024-04-25T09:20:08.867 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.892 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:16615529:::benchmark_data_smithi031_467097_object297:head# 2024-04-25T09:20:08.892 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.917 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:187801dc:::benchmark_data_smithi031_467097_object187:head# 2024-04-25T09:20:08.917 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:08.984 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18919c57:::benchmark_data_smithi031_467097_object230:head# 2024-04-25T09:20:08.984 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.043 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:18bd38b5:::benchmark_data_smithi031_467097_object49:head# 2024-04-25T09:20:09.043 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.093 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:19a5313b:::benchmark_data_smithi031_467097_object134:head# 2024-04-25T09:20:09.093 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.126 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1a967e52:::benchmark_data_smithi031_467097_object385:head# 2024-04-25T09:20:09.126 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.160 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1b51d208:::benchmark_data_smithi031_467097_object55:head# 2024-04-25T09:20:09.160 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.185 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1c76729f:::benchmark_data_smithi031_467097_object4:head# 2024-04-25T09:20:09.185 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.218 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1c8202ce:::benchmark_data_smithi031_467097_object241:head# 2024-04-25T09:20:09.219 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.244 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1d01f931:::benchmark_data_smithi031_467097_object237:head# 2024-04-25T09:20:09.244 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.269 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1d3a89fe:::benchmark_data_smithi031_467097_object113:head# 2024-04-25T09:20:09.269 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.294 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1df332eb:::benchmark_data_smithi031_467097_object73:head# 2024-04-25T09:20:09.294 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.319 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1e97ad6f:::benchmark_data_smithi031_467097_object394:head# 2024-04-25T09:20:09.319 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.352 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1f08b615:::benchmark_data_smithi031_467097_object224:head# 2024-04-25T09:20:09.353 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.377 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:1f197bc1:::benchmark_data_smithi031_467097_object62:head# 2024-04-25T09:20:09.378 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.411 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:212a5056:::benchmark_data_smithi031_467097_object117:head# 2024-04-25T09:20:09.411 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.438 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2220816f:::benchmark_data_smithi031_467097_object390:head# 2024-04-25T09:20:09.438 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.469 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:22326f7a:::benchmark_data_smithi031_467097_object36:head# 2024-04-25T09:20:09.470 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.503 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:22c06904:::benchmark_data_smithi031_467097_object235:head# 2024-04-25T09:20:09.503 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.538 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:252b1027:::benchmark_data_smithi031_467097_object411:head# 2024-04-25T09:20:09.539 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.570 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:25a5f6e6:::benchmark_data_smithi031_467097_object293:head# 2024-04-25T09:20:09.570 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.637 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:25b10ee1:::benchmark_data_smithi031_467097_object188:head# 2024-04-25T09:20:09.637 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.670 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:269bc0eb:::benchmark_data_smithi031_467097_object172:head# 2024-04-25T09:20:09.670 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.703 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:26a4dc22:::benchmark_data_smithi031_467097_object270:head# 2024-04-25T09:20:09.704 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.745 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:26df22c7:::benchmark_data_smithi031_467097_object369:head# 2024-04-25T09:20:09.746 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.787 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2709f079:::benchmark_data_smithi031_467097_object396:head# 2024-04-25T09:20:09.787 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.829 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:278b7215:::benchmark_data_smithi031_467097_object154:head# 2024-04-25T09:20:09.829 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.871 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:279e379a:::benchmark_data_smithi031_467097_object42:head# 2024-04-25T09:20:09.871 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.921 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:284bebcd:::benchmark_data_smithi031_467097_object319:head# 2024-04-25T09:20:09.921 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:09.963 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:296b8d2f:::benchmark_data_smithi031_467097_object383:head# 2024-04-25T09:20:09.963 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.005 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2a3369ea:::benchmark_data_smithi031_467097_object72:head# 2024-04-25T09:20:10.005 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.046 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2abcfdaf:::benchmark_data_smithi031_467097_object138:head# 2024-04-25T09:20:10.046 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.088 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2bbcf18f:::benchmark_data_smithi031_467097_object317:head# 2024-04-25T09:20:10.088 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.130 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2ddd5618:::benchmark_data_smithi031_467097_object283:head# 2024-04-25T09:20:10.130 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.172 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e171fa8:::benchmark_data_smithi031_467097_object289:head# 2024-04-25T09:20:10.172 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.214 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e3b47e8:::benchmark_data_smithi031_467097_object148:head# 2024-04-25T09:20:10.214 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.255 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2e3d6ee6:::benchmark_data_smithi031_467097_object295:head# 2024-04-25T09:20:10.255 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.297 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:2fcfc71f:::benchmark_data_smithi031_467097_object105:head# 2024-04-25T09:20:10.297 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.339 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3074c2ca:::benchmark_data_smithi031_467097_object268:head# 2024-04-25T09:20:10.339 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.380 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:30d72a55:::benchmark_data_smithi031_467097_object150:head# 2024-04-25T09:20:10.381 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.431 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:31f63d35:::benchmark_data_smithi031_467097_object119:head# 2024-04-25T09:20:10.431 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.473 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:32e46cf4:::benchmark_data_smithi031_467097_object278:head# 2024-04-25T09:20:10.473 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.515 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3553be04:::benchmark_data_smithi031_467097_object233:head# 2024-04-25T09:20:10.515 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.556 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:358d4ec2:::benchmark_data_smithi031_467097_object29:head# 2024-04-25T09:20:10.556 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.598 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:35d253e3:::benchmark_data_smithi031_467097_object38:head# 2024-04-25T09:20:10.598 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.665 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:36116e42:::benchmark_data_smithi031_467097_object149:head# 2024-04-25T09:20:10.665 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.723 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:362baf9a:::benchmark_data_smithi031_467097_object415:head# 2024-04-25T09:20:10.724 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.765 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3895cf62:::benchmark_data_smithi031_467097_object306:head# 2024-04-25T09:20:10.765 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.807 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:391dff5e:::benchmark_data_smithi031_467097_object16:head# 2024-04-25T09:20:10.807 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.849 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3924b3fd:::benchmark_data_smithi031_467097_object22:head# 2024-04-25T09:20:10.849 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.882 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:399c8e05:::benchmark_data_smithi031_467097_object342:head# 2024-04-25T09:20:10.883 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:10.916 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3a899b62:::benchmark_data_smithi031_467097_object56:head# 2024-04-25T09:20:10.916 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.000 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b262498:::benchmark_data_smithi031_467097_object213:head# 2024-04-25T09:20:11.000 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.067 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3b64e165:::benchmark_data_smithi031_467097_object211:head# 2024-04-25T09:20:11.067 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.092 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3c71360f:::benchmark_data_smithi031_467097_object169:head# 2024-04-25T09:20:11.092 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.125 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3d086d1b:::benchmark_data_smithi031_467097_object142:head# 2024-04-25T09:20:11.125 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.159 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3e8ccd0d:::benchmark_data_smithi031_467097_object367:head# 2024-04-25T09:20:11.159 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.192 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3eecfee6:::benchmark_data_smithi031_467097_object145:head# 2024-04-25T09:20:11.192 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.226 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:3f13318f:::benchmark_data_smithi031_467097_object382:head# 2024-04-25T09:20:11.226 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.259 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:40762230:::benchmark_data_smithi031_467097_object271:head# 2024-04-25T09:20:11.259 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.292 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:41ba4b29:::benchmark_data_smithi031_467097_object299:head# 2024-04-25T09:20:11.293 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.326 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:42496b2f:::benchmark_data_smithi031_467097_object410:head# 2024-04-25T09:20:11.326 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.359 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:427f0ab0:::benchmark_data_smithi031_467097_object218:head# 2024-04-25T09:20:11.359 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.393 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:43924006:::benchmark_data_smithi031_467097_object248:head# 2024-04-25T09:20:11.393 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.426 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:458620e0:::benchmark_data_smithi031_467097_object40:head# 2024-04-25T09:20:11.426 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.460 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:46af7308:::benchmark_data_smithi031_467097_object143:head# 2024-04-25T09:20:11.460 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.494 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:46b81f28:::benchmark_data_smithi031_467097_object184:head# 2024-04-25T09:20:11.494 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.527 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:47173b92:::benchmark_data_smithi031_467097_object206:head# 2024-04-25T09:20:11.527 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.560 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:498d65aa:::benchmark_data_smithi031_467097_object103:head# 2024-04-25T09:20:11.560 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.602 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:499fa119:::benchmark_data_smithi031_467097_object348:head# 2024-04-25T09:20:11.602 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.643 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4b1439e9:::benchmark_data_smithi031_467097_object156:head# 2024-04-25T09:20:11.644 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.677 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4bd3331c:::benchmark_data_smithi031_467097_object277:head# 2024-04-25T09:20:11.677 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.711 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4d367541:::benchmark_data_smithi031_467097_object137:head# 2024-04-25T09:20:11.711 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.744 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4d6e4159:::benchmark_data_smithi031_467097_object133:head# 2024-04-25T09:20:11.744 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.777 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:4fc94315:::benchmark_data_smithi031_467097_object82:head# 2024-04-25T09:20:11.778 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.811 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:503f3b23:::benchmark_data_smithi031_467097_object407:head# 2024-04-25T09:20:11.811 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.844 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:50fcb57f:::benchmark_data_smithi031_467097_object329:head# 2024-04-25T09:20:11.844 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.878 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51b06ca8:::benchmark_data_smithi031_467097_object146:head# 2024-04-25T09:20:11.878 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.911 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51bcde7e:::benchmark_data_smithi031_467097_object110:head# 2024-04-25T09:20:11.911 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.945 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:51e5bee3:::benchmark_data_smithi031_467097_object349:head# 2024-04-25T09:20:11.945 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:11.978 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:53259d4e:::benchmark_data_smithi031_467097_object276:head# 2024-04-25T09:20:11.978 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.012 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5383104a:::benchmark_data_smithi031_467097_object240:head# 2024-04-25T09:20:12.012 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.045 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:54236b4b:::benchmark_data_smithi031_467097_object227:head# 2024-04-25T09:20:12.045 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.078 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5457fc8f:::benchmark_data_smithi031_467097_object182:head# 2024-04-25T09:20:12.079 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.112 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:548176e3:::benchmark_data_smithi031_467097_object53:head# 2024-04-25T09:20:12.112 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.145 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:54e65407:::benchmark_data_smithi031_467097_object360:head# 2024-04-25T09:20:12.145 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.179 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:554edb45:::benchmark_data_smithi031_467097_object88:head# 2024-04-25T09:20:12.179 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.212 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:55565cdc:::benchmark_data_smithi031_467097_object363:head# 2024-04-25T09:20:12.212 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.246 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:55a8782c:::benchmark_data_smithi031_467097_object226:head# 2024-04-25T09:20:12.246 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.279 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:55b52fde:::benchmark_data_smithi031_467097_object264:head# 2024-04-25T09:20:12.279 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.313 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56096565:::benchmark_data_smithi031_467097_object304:head# 2024-04-25T09:20:12.313 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.346 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5636a188:::benchmark_data_smithi031_467097_object203:head# 2024-04-25T09:20:12.346 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.388 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5678a988:::benchmark_data_smithi031_467097_object136:head# 2024-04-25T09:20:12.388 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.421 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:56d8028f:::benchmark_data_smithi031_467097_object131:head# 2024-04-25T09:20:12.421 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.455 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57ab9ba6:::benchmark_data_smithi031_467097_object76:head# 2024-04-25T09:20:12.455 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.488 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57cb25cd:::benchmark_data_smithi031_467097_object178:head# 2024-04-25T09:20:12.488 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.521 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:57f7ab1b:::benchmark_data_smithi031_467097_object160:head# 2024-04-25T09:20:12.522 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.555 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:592e58e9:::benchmark_data_smithi031_467097_object91:head# 2024-04-25T09:20:12.555 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.588 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:59ca84b1:::benchmark_data_smithi031_467097_object31:head# 2024-04-25T09:20:12.588 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.632 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:59e8d3ef:::benchmark_data_smithi031_467097_object120:head# 2024-04-25T09:20:12.632 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.689 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5a086d35:::benchmark_data_smithi031_467097_object60:head# 2024-04-25T09:20:12.689 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.739 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5ac0437b:::benchmark_data_smithi031_467097_object291:head# 2024-04-25T09:20:12.740 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.781 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5b112f35:::benchmark_data_smithi031_467097_object69:head# 2024-04-25T09:20:12.781 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.823 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5b6c0411:::benchmark_data_smithi031_467097_object167:head# 2024-04-25T09:20:12.823 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5c372236:::benchmark_data_smithi031_467097_object279:head# 2024-04-25T09:20:12.865 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.907 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e0390c8:::benchmark_data_smithi031_467097_object281:head# 2024-04-25T09:20:12.907 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:12.973 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:5e6ccbf9:::benchmark_data_smithi031_467097_object238:head# 2024-04-25T09:20:12.974 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.033 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:602f4300:::benchmark_data_smithi031_467097_object344:head# 2024-04-25T09:20:13.033 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.108 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:60910817:::benchmark_data_smithi031_467097_object159:head# 2024-04-25T09:20:13.108 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.149 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:635636bd:::benchmark_data_smithi031_467097_object15:head# 2024-04-25T09:20:13.149 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.191 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6433df6e:::benchmark_data_smithi031_467097_object337:head# 2024-04-25T09:20:13.191 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.233 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:644bc9ac:::benchmark_data_smithi031_467097_object127:head# 2024-04-25T09:20:13.233 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.275 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:648892a5:::benchmark_data_smithi031_467097_object408:head# 2024-04-25T09:20:13.275 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.316 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:65117d3d:::benchmark_data_smithi031_467097_object401:head# 2024-04-25T09:20:13.317 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.350 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6546cc3d:::benchmark_data_smithi031_467097_object366:head# 2024-04-25T09:20:13.392 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.393 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:66a12924:::benchmark_data_smithi031_467097_object197:head# 2024-04-25T09:20:13.394 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:66fcb472:::benchmark_data_smithi031_467097_object157:head# 2024-04-25T09:20:13.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.475 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6748c1cb:::benchmark_data_smithi031_467097_object403:head# 2024-04-25T09:20:13.475 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.509 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6755413c:::benchmark_data_smithi031_467097_object43:head# 2024-04-25T09:20:13.509 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.542 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:679c2cbe:::benchmark_data_smithi031_467097_object114:head# 2024-04-25T09:20:13.542 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.576 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:67d213ea:::benchmark_data_smithi031_467097_object296:head# 2024-04-25T09:20:13.576 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.609 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:67d43a1b:::benchmark_data_smithi031_467097_object164:head# 2024-04-25T09:20:13.609 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.634 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:698548d2:::benchmark_data_smithi031_467097_object380:head# 2024-04-25T09:20:13.634 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.659 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:69c0acfb:::benchmark_data_smithi031_467097_object364:head# 2024-04-25T09:20:13.660 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.709 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6b6a7d9d:::benchmark_data_smithi031_467097_object413:head# 2024-04-25T09:20:13.710 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.760 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6b7e4d65:::benchmark_data_smithi031_467097_object300:head# 2024-04-25T09:20:13.760 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.793 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6b95753d:::benchmark_data_smithi031_467097_object140:head# 2024-04-25T09:20:13.793 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.852 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ba8f48f:::benchmark_data_smithi031_467097_object135:head# 2024-04-25T09:20:13.852 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.902 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6c22f730:::benchmark_data_smithi031_467097_object368:head# 2024-04-25T09:20:13.902 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:13.969 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ca30cb8:::benchmark_data_smithi031_467097_object34:head# 2024-04-25T09:20:13.969 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.002 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6ce7d6cf:::benchmark_data_smithi031_467097_object395:head# 2024-04-25T09:20:14.003 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.036 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6d8590c1:::benchmark_data_smithi031_467097_object98:head# 2024-04-25T09:20:14.036 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.069 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6e9737ec:::benchmark_data_smithi031_467097_object246:head# 2024-04-25T09:20:14.070 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.103 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:6f70fa66:::benchmark_data_smithi031_467097_object212:head# 2024-04-25T09:20:14.103 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.136 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7097b6e5:::benchmark_data_smithi031_467097_object301:head# 2024-04-25T09:20:14.136 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.170 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7123e38f:::benchmark_data_smithi031_467097_object225:head# 2024-04-25T09:20:14.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.203 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7236a67c:::benchmark_data_smithi031_467097_object162:head# 2024-04-25T09:20:14.203 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.237 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72816d07:::benchmark_data_smithi031_467097_object190:head# 2024-04-25T09:20:14.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.270 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:72b5fb7f:::benchmark_data_smithi031_467097_object155:head# 2024-04-25T09:20:14.270 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.303 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7394813d:::benchmark_data_smithi031_467097_object321:head# 2024-04-25T09:20:14.304 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.337 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:73ec1cbd:::benchmark_data_smithi031_467097_object325:head# 2024-04-25T09:20:14.337 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.370 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:75135d5c:::benchmark_data_smithi031_467097_object351:head# 2024-04-25T09:20:14.371 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.404 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:752ed385:::benchmark_data_smithi031_467097_object254:head# 2024-04-25T09:20:14.404 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:755b566b:::benchmark_data_smithi031_467097_object196:head# 2024-04-25T09:20:14.438 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.471 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7583b304:::benchmark_data_smithi031_467097_object315:head# 2024-04-25T09:20:14.471 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.504 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:76601c60:::benchmark_data_smithi031_467097_object273:head# 2024-04-25T09:20:14.504 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.538 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:774dc987:::benchmark_data_smithi031_467097_object402:head# 2024-04-25T09:20:14.538 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.571 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:78883ce8:::benchmark_data_smithi031_467097_object7:head# 2024-04-25T09:20:14.571 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.605 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:79f84ae2:::benchmark_data_smithi031_467097_object80:head# 2024-04-25T09:20:14.605 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.689 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7a2864bd:::benchmark_data_smithi031_467097_object144:head# 2024-04-25T09:20:14.689 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7a55de80:::benchmark_data_smithi031_467097_object341:head# 2024-04-25T09:20:14.739 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.781 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7a63eab7:::benchmark_data_smithi031_467097_object255:head# 2024-04-25T09:20:14.781 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.814 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7aa2a2b9:::benchmark_data_smithi031_467097_object220:head# 2024-04-25T09:20:14.814 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.847 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7bdc4396:::benchmark_data_smithi031_467097_object11:head# 2024-04-25T09:20:14.848 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.881 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7c05decd:::benchmark_data_smithi031_467097_object106:head# 2024-04-25T09:20:14.881 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.914 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7cd0072f:::benchmark_data_smithi031_467097_object3:head# 2024-04-25T09:20:14.915 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:14.990 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7ce387ef:::benchmark_data_smithi031_467097_object26:head# 2024-04-25T09:20:14.990 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.040 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dd07c2e:::benchmark_data_smithi031_467097_object243:head# 2024-04-25T09:20:15.040 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.090 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7dda6d00:::benchmark_data_smithi031_467097_object340:head# 2024-04-25T09:20:15.091 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7e06874c:::benchmark_data_smithi031_467097_object405:head# 2024-04-25T09:20:15.132 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.174 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7e4a1e32:::benchmark_data_smithi031_467097_object388:head# 2024-04-25T09:20:15.174 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.216 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7e6d8241:::benchmark_data_smithi031_467097_object209:head# 2024-04-25T09:20:15.216 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.258 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7e733df6:::benchmark_data_smithi031_467097_object45:head# 2024-04-25T09:20:15.258 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.299 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:7f9dad66:::benchmark_data_smithi031_467097_object221:head# 2024-04-25T09:20:15.300 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.341 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:80b7356a:::benchmark_data_smithi031_467097_object90:head# 2024-04-25T09:20:15.341 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.383 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:82355ecb:::benchmark_data_smithi031_467097_object261:head# 2024-04-25T09:20:15.383 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.425 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:826dcb52:::benchmark_data_smithi031_467097_object381:head# 2024-04-25T09:20:15.425 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.475 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:832bd708:::benchmark_data_smithi031_467097_object41:head# 2024-04-25T09:20:15.475 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.516 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:834ab85e:::benchmark_data_smithi031_467097_object260:head# 2024-04-25T09:20:15.517 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.559 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8429b72a:::benchmark_data_smithi031_467097_object263:head# 2024-04-25T09:20:15.559 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.600 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:85406674:::benchmark_data_smithi031_467097_object125:head# 2024-04-25T09:20:15.601 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.642 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8764e146:::benchmark_data_smithi031_467097_object107:head# 2024-04-25T09:20:15.642 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.684 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8793b74f:::benchmark_data_smithi031_467097_object50:head# 2024-04-25T09:20:15.684 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.725 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:87947e07:::benchmark_data_smithi031_467097_object365:head# 2024-04-25T09:20:15.726 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.767 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:87be37bd:::benchmark_data_smithi031_467097_object100:head# 2024-04-25T09:20:15.767 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.809 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:87e14a46:::benchmark_data_smithi031_467097_object232:head# 2024-04-25T09:20:15.809 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.851 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8897234b:::benchmark_data_smithi031_467097_object5:head# 2024-04-25T09:20:15.851 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.893 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:88f7b731:::benchmark_data_smithi031_467097_object68:head# 2024-04-25T09:20:15.893 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.926 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:895edafe:::benchmark_data_smithi031_467097_object256:head# 2024-04-25T09:20:15.926 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.959 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:89616ef6:::benchmark_data_smithi031_467097_object251:head# 2024-04-25T09:20:15.960 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:15.993 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8a15ee8a:::benchmark_data_smithi031_467097_object79:head# 2024-04-25T09:20:15.993 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.026 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8a1c6edb:::benchmark_data_smithi031_467097_object54:head# 2024-04-25T09:20:16.027 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.060 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8bd251ab:::benchmark_data_smithi031_467097_object77:head# 2024-04-25T09:20:16.060 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.093 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8c2b6839:::benchmark_data_smithi031_467097_object115:head# 2024-04-25T09:20:16.093 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.127 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8c3c1f2a:::benchmark_data_smithi031_467097_object338:head# 2024-04-25T09:20:16.127 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.160 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8c9bc6bb:::benchmark_data_smithi031_467097_object130:head# 2024-04-25T09:20:16.161 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.185 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8ca24803:::benchmark_data_smithi031_467097_object13:head# 2024-04-25T09:20:16.185 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.210 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8d8b1ec0:::benchmark_data_smithi031_467097_object161:head# 2024-04-25T09:20:16.211 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.244 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8e26ca72:::benchmark_data_smithi031_467097_object58:head# 2024-04-25T09:20:16.244 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.277 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8e68503c:::benchmark_data_smithi031_467097_object350:head# 2024-04-25T09:20:16.277 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.311 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:8eb87e8b:::benchmark_data_smithi031_467097_object347:head# 2024-04-25T09:20:16.311 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.344 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9086a3ff:::benchmark_data_smithi031_467097_object61:head# 2024-04-25T09:20:16.344 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.378 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:91cdb8e7:::benchmark_data_smithi031_467097_object372:head# 2024-04-25T09:20:16.378 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.411 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:92b10e83:::benchmark_data_smithi031_467097_object67:head# 2024-04-25T09:20:16.411 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.445 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:92de843c:::benchmark_data_smithi031_467097_object354:head# 2024-04-25T09:20:16.445 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.478 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:93050ebf:::benchmark_data_smithi031_467097_object309:head# 2024-04-25T09:20:16.478 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.610 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9428fa05:::benchmark_data_smithi031_467097_object250:head# 2024-04-25T09:20:16.610 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.670 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:94839f49:::benchmark_data_smithi031_467097_object358:head# 2024-04-25T09:20:16.670 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.720 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:961116de:::benchmark_data_smithi031_467097_object379:head# 2024-04-25T09:20:16.720 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.754 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:96c6c36c:::benchmark_data_smithi031_467097_object200:head# 2024-04-25T09:20:16.754 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.787 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:973d4c08:::benchmark_data_smithi031_467097_object216:head# 2024-04-25T09:20:16.787 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.821 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:97544e42:::benchmark_data_smithi031_467097_object32:head# 2024-04-25T09:20:16.821 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.854 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9812b2a7:::benchmark_data_smithi031_467097_object373:head# 2024-04-25T09:20:16.854 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.896 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:983c9d6b:::benchmark_data_smithi031_467097_object84:head# 2024-04-25T09:20:16.896 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:16.939 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9848f2b2:::benchmark_data_smithi031_467097_object27:head# 2024-04-25T09:20:16.939 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.024 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9ade755f:::benchmark_data_smithi031_467097_object312:head# 2024-04-25T09:20:17.024 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.097 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9b69017a:::benchmark_data_smithi031_467097_object327:head# 2024-04-25T09:20:17.097 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.131 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9bc5a763:::benchmark_data_smithi031_467097_object174:head# 2024-04-25T09:20:17.131 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.164 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9c08b79c:::benchmark_data_smithi031_467097_object362:head# 2024-04-25T09:20:17.164 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.198 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9d92b034:::benchmark_data_smithi031_467097_object92:head# 2024-04-25T09:20:17.198 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.239 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9deff396:::benchmark_data_smithi031_467097_object20:head# 2024-04-25T09:20:17.240 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.298 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9e06d86e:::benchmark_data_smithi031_467097_object409:head# 2024-04-25T09:20:17.299 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.340 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:9f474749:::benchmark_data_smithi031_467097_object284:head# 2024-04-25T09:20:17.340 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.373 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a20531ba:::benchmark_data_smithi031_467097_object176:head# 2024-04-25T09:20:17.374 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.407 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a2798eba:::benchmark_data_smithi031_467097_object334:head# 2024-04-25T09:20:17.407 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.440 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a3370137:::benchmark_data_smithi031_467097_object118:head# 2024-04-25T09:20:17.440 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.473 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a38ab8b7:::benchmark_data_smithi031_467097_object81:head# 2024-04-25T09:20:17.474 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.507 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a397cde8:::benchmark_data_smithi031_467097_object285:head# 2024-04-25T09:20:17.507 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.540 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a437abdd:::benchmark_data_smithi031_467097_object416:head# 2024-04-25T09:20:17.540 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.574 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a443327d:::benchmark_data_smithi031_467097_object320:head# 2024-04-25T09:20:17.574 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.624 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a500614c:::benchmark_data_smithi031_467097_object229:head# 2024-04-25T09:20:17.624 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.657 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a55c0d56:::benchmark_data_smithi031_467097_object414:head# 2024-04-25T09:20:17.658 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.691 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a6257be7:::benchmark_data_smithi031_467097_object346:head# 2024-04-25T09:20:17.691 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.733 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8bff898:::benchmark_data_smithi031_467097_object215:head# 2024-04-25T09:20:17.733 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.775 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a8e9e624:::benchmark_data_smithi031_467097_object87:head# 2024-04-25T09:20:17.775 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.816 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a96a943e:::benchmark_data_smithi031_467097_object262:head# 2024-04-25T09:20:17.817 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.858 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9a387bd:::benchmark_data_smithi031_467097_object324:head# 2024-04-25T09:20:17.858 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.900 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:a9fa01d4:::benchmark_data_smithi031_467097_object51:head# 2024-04-25T09:20:17.900 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.942 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aa071e7e:::benchmark_data_smithi031_467097_object258:head# 2024-04-25T09:20:17.942 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:17.983 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aa1da460:::benchmark_data_smithi031_467097_object399:head# 2024-04-25T09:20:17.983 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.025 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ab1b18f5:::benchmark_data_smithi031_467097_object46:head# 2024-04-25T09:20:18.025 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.067 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:abb2758c:::benchmark_data_smithi031_467097_object10:head# 2024-04-25T09:20:18.067 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.109 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:acfb2156:::benchmark_data_smithi031_467097_object339:head# 2024-04-25T09:20:18.109 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.150 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aebe48ef:::benchmark_data_smithi031_467097_object95:head# 2024-04-25T09:20:18.151 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.192 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:aed45aba:::benchmark_data_smithi031_467097_object330:head# 2024-04-25T09:20:18.192 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.234 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:af4f7dcb:::benchmark_data_smithi031_467097_object170:head# 2024-04-25T09:20:18.234 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.276 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b08ffde3:::benchmark_data_smithi031_467097_object24:head# 2024-04-25T09:20:18.276 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.318 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b0bb418b:::benchmark_data_smithi031_467097_object323:head# 2024-04-25T09:20:18.318 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.359 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b1859ded:::benchmark_data_smithi031_467097_object28:head# 2024-04-25T09:20:18.359 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.409 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b1fa9eb4:::benchmark_data_smithi031_467097_object64:head# 2024-04-25T09:20:18.410 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.451 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b3dd1b29:::benchmark_data_smithi031_467097_object332:head# 2024-04-25T09:20:18.451 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.493 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b3ee5179:::benchmark_data_smithi031_467097_object111:head# 2024-04-25T09:20:18.493 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.535 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b46e37e5:::benchmark_data_smithi031_467097_object177:head# 2024-04-25T09:20:18.535 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.577 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b4788b5b:::benchmark_data_smithi031_467097_object208:head# 2024-04-25T09:20:18.577 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.630 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b5dea728:::benchmark_data_smithi031_467097_object201:head# 2024-04-25T09:20:18.630 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.702 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b62e4598:::benchmark_data_smithi031_467097_object267:head# 2024-04-25T09:20:18.702 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.735 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b70d191c:::benchmark_data_smithi031_467097_object292:head# 2024-04-25T09:20:18.736 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.769 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b83247a1:::benchmark_data_smithi031_467097_object286:head# 2024-04-25T09:20:18.769 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.802 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:20:18.803 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.836 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:b9f066f9:::benchmark_data_smithi031_467097_object343:head# 2024-04-25T09:20:18.836 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.869 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ba876785:::benchmark_data_smithi031_467097_object252:head# 2024-04-25T09:20:18.869 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.903 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bac5ac8b:::benchmark_data_smithi031_467097_object112:head# 2024-04-25T09:20:18.903 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:18.936 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bafd965c:::benchmark_data_smithi031_467097_object96:head# 2024-04-25T09:20:18.937 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.003 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb11d564:::benchmark_data_smithi031_467097_object288:head# 2024-04-25T09:20:19.003 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.071 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb672e07:::benchmark_data_smithi031_467097_object361:head# 2024-04-25T09:20:19.071 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.104 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bb6db15c:::benchmark_data_smithi031_467097_object355:head# 2024-04-25T09:20:19.104 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.137 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bc34dff4:::benchmark_data_smithi031_467097_object185:head# 2024-04-25T09:20:19.137 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.171 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bc931565:::benchmark_data_smithi031_467097_object158:head# 2024-04-25T09:20:19.171 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.204 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bd41ee77:::benchmark_data_smithi031_467097_object266:head# 2024-04-25T09:20:19.204 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.238 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:be83025a:::benchmark_data_smithi031_467097_object397:head# 2024-04-25T09:20:19.238 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.271 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:be915db4:::benchmark_data_smithi031_467097_object290:head# 2024-04-25T09:20:19.271 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.304 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bea2df34:::benchmark_data_smithi031_467097_object121:head# 2024-04-25T09:20:19.305 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.338 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bf1f6802:::benchmark_data_smithi031_467097_object219:head# 2024-04-25T09:20:19.338 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.371 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:bf5dd63e:::benchmark_data_smithi031_467097_object109:head# 2024-04-25T09:20:19.372 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.405 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c0fb5b88:::benchmark_data_smithi031_467097_object205:head# 2024-04-25T09:20:19.405 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.446 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c14cc90b:::benchmark_data_smithi031_467097_object94:head# 2024-04-25T09:20:19.447 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.480 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c157d7c3:::benchmark_data_smithi031_467097_object19:head# 2024-04-25T09:20:19.480 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.514 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c1bf13c4:::benchmark_data_smithi031_467097_object37:head# 2024-04-25T09:20:19.514 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.556 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c1d86d32:::benchmark_data_smithi031_467097_object269:head# 2024-04-25T09:20:19.556 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.589 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c1e3f260:::benchmark_data_smithi031_467097_object275:head# 2024-04-25T09:20:19.589 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.622 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c27f1e0a:::benchmark_data_smithi031_467097_object375:head# 2024-04-25T09:20:19.623 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.656 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c2ce63b1:::benchmark_data_smithi031_467097_object326:head# 2024-04-25T09:20:19.656 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.689 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c3c19702:::benchmark_data_smithi031_467097_object147:head# 2024-04-25T09:20:19.690 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.723 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c3f54765:::benchmark_data_smithi031_467097_object48:head# 2024-04-25T09:20:19.723 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.756 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c4049b17:::benchmark_data_smithi031_467097_object9:head# 2024-04-25T09:20:19.756 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.790 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c4546446:::benchmark_data_smithi031_467097_object234:head# 2024-04-25T09:20:19.790 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.823 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c46b9f43:::benchmark_data_smithi031_467097_object75:head# 2024-04-25T09:20:19.823 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.857 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c52ac604:::benchmark_data_smithi031_467097_object199:head# 2024-04-25T09:20:19.857 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.890 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c603b7d6:::benchmark_data_smithi031_467097_object198:head# 2024-04-25T09:20:19.890 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.924 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c63fe32b:::benchmark_data_smithi031_467097_object244:head# 2024-04-25T09:20:19.924 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:19.974 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c7da009d:::benchmark_data_smithi031_467097_object23:head# 2024-04-25T09:20:19.974 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.007 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c8d1401f:::benchmark_data_smithi031_467097_object124:head# 2024-04-25T09:20:20.007 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.041 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c9ee5ab7:::benchmark_data_smithi031_467097_object253:head# 2024-04-25T09:20:20.041 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.074 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:c9ff2023:::benchmark_data_smithi031_467097_object318:head# 2024-04-25T09:20:20.074 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.132 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca11222e:::benchmark_data_smithi031_467097_object245:head# 2024-04-25T09:20:20.133 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.166 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca467a5a:::benchmark_data_smithi031_467097_object298:head# 2024-04-25T09:20:20.166 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.199 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca504494:::benchmark_data_smithi031_467097_object328:head# 2024-04-25T09:20:20.200 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.233 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca518a1d:::benchmark_data_smithi031_467097_object108:head# 2024-04-25T09:20:20.233 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.266 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ca76242d:::benchmark_data_smithi031_467097_object202:head# 2024-04-25T09:20:20.266 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.300 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cabf5cf4:::benchmark_data_smithi031_467097_object308:head# 2024-04-25T09:20:20.300 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.333 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cb59f22b:::benchmark_data_smithi031_467097_object30:head# 2024-04-25T09:20:20.333 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.367 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cb72c784:::benchmark_data_smithi031_467097_object377:head# 2024-04-25T09:20:20.367 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.400 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cc2e932a:::benchmark_data_smithi031_467097_object265:head# 2024-04-25T09:20:20.400 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.433 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cd0c4b72:::benchmark_data_smithi031_467097_object376:head# 2024-04-25T09:20:20.434 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.467 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ce6786b9:::benchmark_data_smithi031_467097_object44:head# 2024-04-25T09:20:20.467 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.500 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ced3112b:::benchmark_data_smithi031_467097_object242:head# 2024-04-25T09:20:20.501 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.534 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cf735205:::benchmark_data_smithi031_467097_object168:head# 2024-04-25T09:20:20.534 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.576 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:cf81d81f:::benchmark_data_smithi031_467097_object1:head# 2024-04-25T09:20:20.576 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.621 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d0c5feda:::benchmark_data_smithi031_467097_object210:head# 2024-04-25T09:20:20.622 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.693 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d0cea92e:::benchmark_data_smithi031_467097_object303:head# 2024-04-25T09:20:20.693 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.743 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d14da95c:::benchmark_data_smithi031_467097_object307:head# 2024-04-25T09:20:20.743 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.785 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d165a956:::benchmark_data_smithi031_467097_object89:head# 2024-04-25T09:20:20.785 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.826 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d17ee123:::benchmark_data_smithi031_467097_object287:head# 2024-04-25T09:20:20.827 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.868 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d1d9b532:::benchmark_data_smithi031_467097_object57:head# 2024-04-25T09:20:20.868 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.910 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d1e2873f:::benchmark_data_smithi031_467097_object129:head# 2024-04-25T09:20:20.910 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:20.977 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d2af251c:::benchmark_data_smithi031_467097_object294:head# 2024-04-25T09:20:20.977 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.032 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d3d8c1e5:::benchmark_data_smithi031_467097_object228:head# 2024-04-25T09:20:21.032 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.136 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d47becaf:::benchmark_data_smithi031_467097_object398:head# 2024-04-25T09:20:21.136 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.169 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d4d48e4c:::benchmark_data_smithi031_467097_object352:head# 2024-04-25T09:20:21.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.203 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d4f0a3d2:::benchmark_data_smithi031_467097_object384:head# 2024-04-25T09:20:21.203 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.236 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d4fabdb8:::benchmark_data_smithi031_467097_object314:head# 2024-04-25T09:20:21.237 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.270 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d5252a4b:::benchmark_data_smithi031_467097_object322:head# 2024-04-25T09:20:21.270 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.303 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d6826dbd:::benchmark_data_smithi031_467097_object104:head# 2024-04-25T09:20:21.304 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.337 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:d9d97115:::benchmark_data_smithi031_467097_object222:head# 2024-04-25T09:20:21.337 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.370 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:da9a2295:::benchmark_data_smithi031_467097_object116:head# 2024-04-25T09:20:21.370 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.404 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:daa1228a:::benchmark_data_smithi031_467097_object374:head# 2024-04-25T09:20:21.404 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.437 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dabcbbca:::benchmark_data_smithi031_467097_object25:head# 2024-04-25T09:20:21.437 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.471 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dcbce9ae:::benchmark_data_smithi031_467097_object302:head# 2024-04-25T09:20:21.471 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.504 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dcebadbc:::benchmark_data_smithi031_467097_object86:head# 2024-04-25T09:20:21.504 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.537 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd8b0387:::benchmark_data_smithi031_467097_object194:head# 2024-04-25T09:20:21.538 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.571 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:dd8c5adf:::benchmark_data_smithi031_467097_object163:head# 2024-04-25T09:20:21.571 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.604 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ddb4c7d1:::benchmark_data_smithi031_467097_object378:head# 2024-04-25T09:20:21.604 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.671 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:de11411f:::benchmark_data_smithi031_467097_object99:head# 2024-04-25T09:20:21.671 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.705 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:deccc71b:::benchmark_data_smithi031_467097_object195:head# 2024-04-25T09:20:21.705 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.738 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ded9d547:::benchmark_data_smithi031_467097_object357:head# 2024-04-25T09:20:21.738 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.772 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:df4c3bad:::benchmark_data_smithi031_467097_object132:head# 2024-04-25T09:20:21.772 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.814 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:df94ff48:::benchmark_data_smithi031_467097_object52:head# 2024-04-25T09:20:21.814 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.847 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e01aac02:::benchmark_data_smithi031_467097_object217:head# 2024-04-25T09:20:21.847 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.880 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e023c0bf:::benchmark_data_smithi031_467097_object2:head# 2024-04-25T09:20:21.881 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.914 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e03c6ba0:::benchmark_data_smithi031_467097_object387:head# 2024-04-25T09:20:21.914 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.948 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e0404815:::benchmark_data_smithi031_467097_object6:head# 2024-04-25T09:20:21.948 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:21.981 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e158c29b:::benchmark_data_smithi031_467097_object35:head# 2024-04-25T09:20:21.981 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.014 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e1ade39b:::benchmark_data_smithi031_467097_object21:head# 2024-04-25T09:20:22.015 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.048 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e2c51ff4:::benchmark_data_smithi031_467097_object85:head# 2024-04-25T09:20:22.048 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.081 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e2cafd8f:::benchmark_data_smithi031_467097_object223:head# 2024-04-25T09:20:22.081 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.115 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e314ee7c:::benchmark_data_smithi031_467097_object74:head# 2024-04-25T09:20:22.115 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.148 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e39ec5ea:::benchmark_data_smithi031_467097_object83:head# 2024-04-25T09:20:22.148 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.181 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e48acdbc:::benchmark_data_smithi031_467097_object65:head# 2024-04-25T09:20:22.182 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.215 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e49798ea:::benchmark_data_smithi031_467097_object207:head# 2024-04-25T09:20:22.215 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.248 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e677fc79:::benchmark_data_smithi031_467097_object173:head# 2024-04-25T09:20:22.248 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.282 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e737ca82:::benchmark_data_smithi031_467097_object47:head# 2024-04-25T09:20:22.282 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.315 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e7522751:::benchmark_data_smithi031_467097_object186:head# 2024-04-25T09:20:22.316 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.349 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e7617382:::benchmark_data_smithi031_467097_object404:head# 2024-04-25T09:20:22.349 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.382 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e85963b8:::benchmark_data_smithi031_467097_object310:head# 2024-04-25T09:20:22.382 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.416 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:e8924085:::benchmark_data_smithi031_467097_object17:head# 2024-04-25T09:20:22.416 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.449 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:eac421ca:::benchmark_data_smithi031_467097_object14:head# 2024-04-25T09:20:22.449 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.483 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:eb26f6d7:::benchmark_data_smithi031_467097_object356:head# 2024-04-25T09:20:22.483 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.516 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ee0210d1:::benchmark_data_smithi031_467097_object70:head# 2024-04-25T09:20:22.516 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.575 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ee410e68:::benchmark_data_smithi031_467097_object180:head# 2024-04-25T09:20:22.575 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.608 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ef2b8b1f:::benchmark_data_smithi031_467097_object101:head# 2024-04-25T09:20:22.608 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.658 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f26dffcd:::benchmark_data_smithi031_467097_object204:head# 2024-04-25T09:20:22.659 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.708 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4c7ad45:::benchmark_data_smithi031_467097_object66:head# 2024-04-25T09:20:22.708 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.742 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f4e38a5e:::benchmark_data_smithi031_467097_object39:head# 2024-04-25T09:20:22.742 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.775 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f5ae7d63:::benchmark_data_smithi031_467097_object274:head# 2024-04-25T09:20:22.776 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.809 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f5cb5ac6:::benchmark_data_smithi031_467097_object78:head# 2024-04-25T09:20:22.809 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.842 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f62e2160:::benchmark_data_smithi031_467097_object152:head# 2024-04-25T09:20:22.842 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.875 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f655c7e6:::benchmark_data_smithi031_467097_object141:head# 2024-04-25T09:20:22.876 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.909 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f65a695f:::benchmark_data_smithi031_467097_object63:head# 2024-04-25T09:20:22.910 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:22.968 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f6fdedb6:::benchmark_data_smithi031_467097_object122:head# 2024-04-25T09:20:22.968 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.018 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f7516ecf:::benchmark_data_smithi031_467097_object391:head# 2024-04-25T09:20:23.018 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.085 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f7d922b0:::benchmark_data_smithi031_467097_object259:head# 2024-04-25T09:20:23.085 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.127 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f7e08e1e:::benchmark_data_smithi031_467097_object386:head# 2024-04-25T09:20:23.127 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.170 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f7e6d340:::benchmark_data_smithi031_467097_object0:head# 2024-04-25T09:20:23.170 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.211 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f8530af6:::benchmark_data_smithi031_467097_object97:head# 2024-04-25T09:20:23.211 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.252 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f86e4a3f:::benchmark_data_smithi031_467097_object179:head# 2024-04-25T09:20:23.253 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.294 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f96e7e7a:::benchmark_data_smithi031_467097_object331:head# 2024-04-25T09:20:23.294 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.336 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:f9d769e5:::benchmark_data_smithi031_467097_object412:head# 2024-04-25T09:20:23.336 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.378 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fa441986:::benchmark_data_smithi031_467097_object8:head# 2024-04-25T09:20:23.378 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.420 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:faa9086e:::benchmark_data_smithi031_467097_object93:head# 2024-04-25T09:20:23.420 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.461 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:faab5dd1:::benchmark_data_smithi031_467097_object102:head# 2024-04-25T09:20:23.461 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.503 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fac1e97a:::benchmark_data_smithi031_467097_object335:head# 2024-04-25T09:20:23.503 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.545 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb0c180e:::benchmark_data_smithi031_467097_object126:head# 2024-04-25T09:20:23.545 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.586 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fb9f7ad8:::benchmark_data_smithi031_467097_object280:head# 2024-04-25T09:20:23.586 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.628 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fce0f0a2:::benchmark_data_smithi031_467097_object247:head# 2024-04-25T09:20:23.628 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.662 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fddbf60a:::benchmark_data_smithi031_467097_object371:head# 2024-04-25T09:20:23.662 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.695 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fdfa9640:::benchmark_data_smithi031_467097_object316:head# 2024-04-25T09:20:23.695 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.729 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:fe7e28e5:::benchmark_data_smithi031_467097_object392:head# 2024-04-25T09:20:23.729 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.762 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:feb36f49:::benchmark_data_smithi031_467097_object282:head# 2024-04-25T09:20:23.762 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.796 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:feb403bf:::benchmark_data_smithi031_467097_object151:head# 2024-04-25T09:20:23.796 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.829 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ff3a0c02:::benchmark_data_smithi031_467097_object189:head# 2024-04-25T09:20:23.829 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.863 INFO:tasks.workunit.client.0.smithi031.stdout:Write #1:ffbbf3bd:::benchmark_data_smithi031_467097_object406:head# 2024-04-25T09:20:23.863 INFO:tasks.workunit.client.0.smithi031.stdout:snapset 0=[]:{} 2024-04-25T09:20:23.897 INFO:tasks.workunit.client.0.smithi031.stdout:write_pg epoch 9 info 1.0( v 9'418 (0'0,9'418] local-lis/les=7/8 n=418 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:20:23.938 INFO:tasks.workunit.client.0.smithi031.stdout:Import successful 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:20:24.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:24.479 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:20:24.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:20:24.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:20:24.482 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:20:24.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=81141e49-6680-4da7-b158-e94256f9a1d7 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.443986/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:20:24.482 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:20:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:20:24.483 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:20:24.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:20:24.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:20:24.519 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:24.529+0000 7eff63a297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:24.522 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:24.533+0000 7eff63a297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:24.523 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:24.533+0000 7eff63a297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:25.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:20:25.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:20:25.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:20:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:20:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:20:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:25.031 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:20:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:20:25.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:25.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:25.077 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:25.089+0000 7eff63a297c0 -1 Falling back to public interface 2024-04-25T09:20:25.496 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:26.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:26.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:26.498 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:20:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:20:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:26.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:26.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:27.939 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:27.949+0000 7eff63a297c0 -1 osd.0 18 log_to_monitors true 2024-04-25T09:20:27.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:27.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:27.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:20:27.981 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:20:27.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:27.981 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:28.413 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 21 up_thru 16 down_at 19 last_clean_interval [12,18) [v2:127.0.0.1:6802/3776330666,v1:127.0.0.1:6803/3776330666] [v2:127.0.0.1:6804/3776330666,v1:127.0.0.1:6805/3776330666] exists,up 7e17a2f7-7821-4592-a394-a058baf89f2d 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:20:28.414 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:20:28.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:20:28.415 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:20:28.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:20:28.415 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:20:28.416 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:20:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:20:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:20:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:20:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:20:28.528 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:20:28.528 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:20:28.994 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:20:28.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:20:28.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:28.995 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:20:29.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313219 2024-04-25T09:20:29.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313219 2024-04-25T09:20:29.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-90194313219' 2024-04-25T09:20:29.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:29.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-90194313219 2024-04-25T09:20:29.159 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:29.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:20:29.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:29.161 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-90194313219 2024-04-25T09:20:29.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313219 2024-04-25T09:20:29.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 90194313219' 2024-04-25T09:20:29.162 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 90194313219 2024-04-25T09:20:29.163 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:29.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 90194313219 2024-04-25T09:20:29.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:20:30.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:20:30.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:31.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313219 -lt 90194313219 2024-04-25T09:20:31.059 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:20:31.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:31.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:31.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:20:31.629 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:20:32.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:20:32.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:20:32.033 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:32.034 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:32.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:20:32.578 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:20:32.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:20:32.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:20:32.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:20:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:20:32.888 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:20:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:20:32.889 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:20:32.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:20:32.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:20:32.892 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:20:32.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:20:32.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:20:32.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:20:32.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:20:32.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:20:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:20:32.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:20:32.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:20:32.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:32.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:20:32.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:20:32.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:20:32.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:20:32.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:20:32.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/pg-split-merge 0 2024-04-25T09:20:32.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:20:32.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:20:32.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:20:32.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:20:32.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:20:32.916 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:20:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:20:32.917 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:20:32.918 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:20:32.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:20:32.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:20:32.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:20:32.920 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:20:32.920 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:20:32.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:20:32.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:20:32.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:20:32.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:32.924 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.443986 2024-04-25T09:20:32.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.443986 2024-04-25T09:20:32.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:20:32.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:20:32.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:20:32.926 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:20:32.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:20:32.984 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-04-25T09:20:32.984 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-04-25T09:20:33.040 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:20:33.045 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:20:33.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:20:33.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:20:33.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repeer-on-acting-back 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-04-25T09:20:33.046 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-04-25T09:20:33.047 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-04-25T09:20:33.048 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-04-25T09:20:33.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none ' 2024-04-25T09:20:33.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T09:20:33.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:20:33.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:20:33.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-04-25T09:20:33.049 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:20:33.052 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:20:33.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:20:33.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:20:33.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:20:33.054 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:20:33.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:20:33.055 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:20:33.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:20:33.056 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:20:33.056 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:20:33.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:20:33.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:20:33.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:20:33.058 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:20:33.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:20:33.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:20:33.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:20:33.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:20:33.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:33.061 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:33.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470537 2024-04-25T09:20:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:20:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:20:33.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-04-25T09:20:33.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:20:33.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:33.064 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:33.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.470537 2024-04-25T09:20:33.066 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:20:33.066 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-04-25T09:20:33.067 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-04-25T09:20:33.068 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:20:33.069 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-04-25T09:20:33.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-04-25T09:20:33.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:20:33.544 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:33.545 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:33.545 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:33.545 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:33.545 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:33.546 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:20:33.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:20:33.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:20:33.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:20:33.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:20:33.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:20:33.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:20:33.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:20:33.759 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470537/ceph-mon.a.asok 2024-04-25T09:20:33.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:20:33.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470537/ceph-mon.a.asok config get fsid 2024-04-25T09:20:33.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:20:33.878 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:20:33.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:20:33.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:20:33.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.470537/ceph-mon.a.asok 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:20:33.880 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.470537/ceph-mon.a.asok config get mon_host 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-04-25T09:20:33.995 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:20:34.922 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:20:34.922 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:34.922 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:34.923 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:34.923 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:34.923 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:34.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:20:34.923 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:20:34.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:20:34.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:20:34.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-04-25T09:20:34.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:20:34.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:20:34.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:20:34.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:34.990 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:20:34.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:20:34.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-04-25T09:20:34.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:20:34.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a4431bef-77ca-420f-99c1-aa7669a835af 2024-04-25T09:20:34.992 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 a4431bef-77ca-420f-99c1-aa7669a835af 2024-04-25T09:20:34.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a4431bef-77ca-420f-99c1-aa7669a835af' 2024-04-25T09:20:34.992 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:20:35.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjICpm1fDCARAAuN5Wj6Wqd/C4KrsX2xtiaQ== 2024-04-25T09:20:35.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjICpm1fDCARAAuN5Wj6Wqd/C4KrsX2xtiaQ=="}' 2024-04-25T09:20:35.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a4431bef-77ca-420f-99c1-aa7669a835af -i td/repeer-on-acting-back/0/new.json 2024-04-25T09:20:35.309 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:20:35.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-04-25T09:20:35.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjICpm1fDCARAAuN5Wj6Wqd/C4KrsX2xtiaQ== --osd-uuid a4431bef-77ca-420f-99c1-aa7669a835af 2024-04-25T09:20:35.376 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.388+0000 7f5791a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:35.379 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.388+0000 7f5791a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:35.380 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.392+0000 7f5791a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:35.381 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.392+0000 7f5791a307c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:20:35.381 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.392+0000 7f5791a307c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:20:35.381 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.392+0000 7f5791a307c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:20:35.418 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:35.428+0000 7f5791a307c0 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-04-25T09:20:41.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-04-25T09:20:41.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:20:41.563 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:20:41.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:20:41.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:20:42.177 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:20:42.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:20:42.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:20:42.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:20:42.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:20:42.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:20:42.218 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:42.228+0000 7f9bab9f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:42.250 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:42.260+0000 7f9bab9f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:42.251 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:42.260+0000 7f9bab9f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:42.665 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:20:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:20:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:20:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:20:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:20:42.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:42.666 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:20:42.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:20:42.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:42.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:42.809 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:42.820+0000 7f9bab9f07c0 -1 Falling back to public interface 2024-04-25T09:20:43.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:44.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:44.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:44.183 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:20:44.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:20:44.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:44.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:44.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:45.325 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:20:45.336+0000 7f9bab9f07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:20:45.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:45.669 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:20:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:20:45.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:45.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:46.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:47.350 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:47.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:47.351 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:20:47.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:20:47.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:47.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:47.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:48.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:48.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:48.799 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:20:48.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:20:48.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:48.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:49.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:50.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:50.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:50.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:20:50.247 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:20:50.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:50.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:50.686 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:20:51.689 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:20:51.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:51.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:52.229 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:53.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:53.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:53.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:20:53.232 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:20:53.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:53.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:53.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:54.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:54.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:54.673 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:20:54.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:20:54.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:54.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:55.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:56.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:56.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:56.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:20:56.218 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:20:56.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:56.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:56.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:57.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:57.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:57.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:20:57.661 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:20:57.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:57.661 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:58.382 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:59.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:59.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:59.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:20:59.384 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:20:59.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:59.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:00.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:01.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:01.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:01.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:21:01.701 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:21:01.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:01.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:02.824 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:03.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:03.826 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:21:03.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:03.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:21:03.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:03.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:04.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:05.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:05.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:05.737 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:21:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:21:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:05.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:06.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:07.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:07.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:07.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:21:07.790 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:21:07.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:07.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:08.370 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/496952644,v1:127.0.0.1:6803/496952644] [v2:127.0.0.1:6804/496952644,v1:127.0.0.1:6805/496952644] exists,up a4431bef-77ca-420f-99c1-aa7669a835af 2024-04-25T09:21:08.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:08.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:21:08.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:21:08.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:21:08.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-04-25T09:21:08.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:21:08.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1f7ef89-70a5-42ba-8eed-a8a3ab0e461c 2024-04-25T09:21:08.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d1f7ef89-70a5-42ba-8eed-a8a3ab0e461c' 2024-04-25T09:21:08.377 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 d1f7ef89-70a5-42ba-8eed-a8a3ab0e461c 2024-04-25T09:21:08.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:21:08.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEICpmZX2/GBAAnIND4CdC7S4VAhq7KUUBag== 2024-04-25T09:21:08.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEICpmZX2/GBAAnIND4CdC7S4VAhq7KUUBag=="}' 2024-04-25T09:21:08.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1f7ef89-70a5-42ba-8eed-a8a3ab0e461c -i td/repeer-on-acting-back/1/new.json 2024-04-25T09:21:08.906 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:21:08.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-04-25T09:21:08.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEICpmZX2/GBAAnIND4CdC7S4VAhq7KUUBag== --osd-uuid d1f7ef89-70a5-42ba-8eed-a8a3ab0e461c 2024-04-25T09:21:08.965 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:08.976+0000 7fe2a23557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:08.970 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:08.976+0000 7fe2a23557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:08.970 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:08.980+0000 7fe2a23557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:08.970 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:08.980+0000 7fe2a23557c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:21:08.970 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:08.980+0000 7fe2a23557c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:21:08.971 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:08.980+0000 7fe2a23557c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:21:09.010 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:09.024+0000 7fe2a23557c0 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-04-25T09:21:15.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-04-25T09:21:15.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:21:15.137 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:21:15.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:21:15.137 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:21:15.711 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:21:15.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:21:15.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:21:15.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:21:15.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:21:15.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:21:15.750 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:15.764+0000 7ffa4dc2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:15.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:15.772+0000 7ffa4dc2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:15.759 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:15.772+0000 7ffa4dc2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:16.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:21:16.200 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:21:16.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:16.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:16.313 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:16.328+0000 7ffa4dc2e7c0 -1 Falling back to public interface 2024-04-25T09:21:16.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:17.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:17.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:17.693 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:21:17.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:21:17.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:17.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:18.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:18.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:18.812+0000 7ffa4dc2e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:21:19.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:19.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:19.156 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:21:19.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:21:19.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:19.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:19.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:20.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:20.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:20.631 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:21:20.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:21:20.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:20.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:21.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:22.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:22.163 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:22.163 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:21:22.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:21:22.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:22.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:22.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:23.735 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:23.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:23.736 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:21:23.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:21:23.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:23.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:24.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:25.321 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:21:25.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:21:25.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:25.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:25.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:26.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:26.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:26.814 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:21:26.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:21:26.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:26.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:27.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:28.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:28.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:28.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:21:28.294 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:21:28.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:28.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:28.786 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:29.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:29.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:29.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:21:29.788 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:21:29.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:29.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:30.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:31.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:31.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:31.297 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:21:31.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:21:31.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:31.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:32.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:33.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:33.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:33.296 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:21:33.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:21:33.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:33.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:34.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:35.076 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:35.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:35.077 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:21:35.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:21:35.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:35.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:35.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:36.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:36.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:36.803 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:21:36.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:21:36.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:36.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:37.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:38.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:38.537 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:38.538 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:21:38.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:21:38.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:38.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:39.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:40.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:40.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:40.531 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:21:40.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:21:40.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:40.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:41.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:42.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:42.480 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:21:42.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:42.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:21:42.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:42.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:42.926 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1080880757,v1:127.0.0.1:6811/1080880757] [v2:127.0.0.1:6812/1080880757,v1:127.0.0.1:6813/1080880757] exists,up d1f7ef89-70a5-42ba-8eed-a8a3ab0e461c 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:21:42.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:21:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:21:42.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:21:42.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d79f2943-4567-402a-ae88-1731af2bad2d 2024-04-25T09:21:42.937 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 d79f2943-4567-402a-ae88-1731af2bad2d 2024-04-25T09:21:42.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d79f2943-4567-402a-ae88-1731af2bad2d' 2024-04-25T09:21:42.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:21:42.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmICpmTmMXOhAAsYJBBcd5jlYVUJpoKeFOWQ== 2024-04-25T09:21:42.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmICpmTmMXOhAAsYJBBcd5jlYVUJpoKeFOWQ=="}' 2024-04-25T09:21:42.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d79f2943-4567-402a-ae88-1731af2bad2d -i td/repeer-on-acting-back/2/new.json 2024-04-25T09:21:43.489 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:21:43.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-04-25T09:21:43.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmICpmTmMXOhAAsYJBBcd5jlYVUJpoKeFOWQ== --osd-uuid d79f2943-4567-402a-ae88-1731af2bad2d 2024-04-25T09:21:43.548 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.563+0000 7f3c3dbd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:43.577 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.591+0000 7f3c3dbd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:43.579 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.591+0000 7f3c3dbd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:43.579 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.595+0000 7f3c3dbd47c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:21:43.580 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.595+0000 7f3c3dbd47c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:21:43.580 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.595+0000 7f3c3dbd47c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:21:43.607 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:43.623+0000 7f3c3dbd47c0 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-04-25T09:21:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-04-25T09:21:49.645 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:21:49.647 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:21:49.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:21:49.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:21:50.267 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:21:50.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:21:50.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:21:50.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:21:50.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:21:50.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:21:50.307 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:50.319+0000 7f40b2f717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:50.309 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:50.323+0000 7f40b2f717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:50.310 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:50.323+0000 7f40b2f717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:50.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:21:50.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:21:50.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:21:50.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:21:50.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:21:50.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:50.700 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:21:50.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:21:50.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:50.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:50.881 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:50.895+0000 7f40b2f717c0 -1 Falling back to public interface 2024-04-25T09:21:51.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:52.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:21:52.185 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:21:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:52.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:52.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:53.437 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:21:53.451+0000 7f40b2f717c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:21:53.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:53.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:53.626 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:21:53.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:21:53.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:53.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:54.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:55.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:55.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:55.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:21:55.103 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:21:55.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:55.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:55.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:56.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:56.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:56.655 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:21:56.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:21:56.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:56.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:57.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:58.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:58.100 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:21:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:21:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:58.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:58.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:59.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:59.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:59.717 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:21:59.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:21:59.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:59.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:00.245 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:01.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:01.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:01.247 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:22:01.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:22:01.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:01.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:01.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:02.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:02.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:02.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:22:02.701 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:22:02.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:02.701 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:03.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:04.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:04.274 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:04.274 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:22:04.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:22:04.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:04.275 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:04.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:05.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:05.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:05.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:22:05.739 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:22:05.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:05.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:06.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:07.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:07.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:07.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:22:07.610 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:22:07.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:07.610 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:08.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:09.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:09.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:09.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:22:09.423 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:22:09.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:09.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:10.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:11.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:22:11.177 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:22:11.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:11.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:11.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:12.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:12.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:12.893 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:22:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:22:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:12.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:13.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:14.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:14.659 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:14.660 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:22:14.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:22:14.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:14.660 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:15.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:16.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:16.599 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:16.600 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:22:16.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:22:16.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:16.600 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:17.109 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1139527067,v1:127.0.0.1:6819/1139527067] [v2:127.0.0.1:6820/1139527067,v1:127.0.0.1:6821/1139527067] exists,up d79f2943-4567-402a-ae88-1731af2bad2d 2024-04-25T09:22:17.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:22:17.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:22:17.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:22:17.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:22:17.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-04-25T09:22:17.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:22:17.113 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:22:17.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-04-25T09:22:17.115 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:22:17.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ec43e422-661b-4045-b207-f5bdeced86bd 2024-04-25T09:22:17.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ec43e422-661b-4045-b207-f5bdeced86bd' 2024-04-25T09:22:17.116 INFO:tasks.workunit.client.0.smithi031.stdout:add osd3 ec43e422-661b-4045-b207-f5bdeced86bd 2024-04-25T09:22:17.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:22:17.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJICpmGi5nCRAA0ufy58iGJIL/mdrU3qRTFA== 2024-04-25T09:22:17.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJICpmGi5nCRAA0ufy58iGJIL/mdrU3qRTFA=="}' 2024-04-25T09:22:17.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ec43e422-661b-4045-b207-f5bdeced86bd -i td/repeer-on-acting-back/3/new.json 2024-04-25T09:22:17.749 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:22:17.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-04-25T09:22:17.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJICpmGi5nCRAA0ufy58iGJIL/mdrU3qRTFA== --osd-uuid ec43e422-661b-4045-b207-f5bdeced86bd 2024-04-25T09:22:17.803 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.815+0000 7f247a04a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:17.822 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.835+0000 7f247a04a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:17.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.839+0000 7f247a04a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:17.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.839+0000 7f247a04a7c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:22:17.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.839+0000 7f247a04a7c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:22:17.824 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.839+0000 7f247a04a7c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:22:17.922 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:17.935+0000 7f247a04a7c0 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-04-25T09:22:23.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-04-25T09:22:23.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:22:23.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:22:23.968 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd3 key to auth repository 2024-04-25T09:22:23.968 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:22:24.595 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.3 2024-04-25T09:22:24.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:22:24.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:22:24.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:22:24.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:22:24.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:22:24.627 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:24.639+0000 7fd866af17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:24.645 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:24.659+0000 7fd866af17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:24.646 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:24.659+0000 7fd866af17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:25.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:22:25.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:22:25.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:22:25.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:22:25.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:22:25.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:25.059 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:22:25.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:22:25.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:25.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:25.213 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:25.227+0000 7fd866af17c0 -1 Falling back to public interface 2024-04-25T09:22:25.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:26.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:26.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:26.481 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:22:26.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:22:26.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:26.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:26.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:27.786 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:27.799+0000 7fd866af17c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:22:27.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:27.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:27.916 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:22:27.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:22:27.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:27.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:28.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:29.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:29.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:29.570 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:22:29.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:22:29.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:29.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:30.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:31.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:31.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:31.061 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:22:31.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:22:31.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:31.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:31.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:32.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:32.502 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:32.502 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:22:32.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:22:32.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:32.503 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:32.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:34.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:34.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:34.001 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:22:34.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:22:34.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:34.001 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:34.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:35.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:35.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:35.465 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:22:35.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:22:35.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:35.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:36.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:37.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:37.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:37.031 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:22:37.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:22:37.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:37.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:37.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:38.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:38.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:38.493 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:22:38.493 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:22:38.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:38.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:38.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:39.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:39.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:39.972 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:22:39.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:22:39.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:39.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:40.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:41.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:41.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:41.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:22:41.425 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:22:41.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:41.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:42.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:43.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:43.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:43.606 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:22:43.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:22:43.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:43.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:44.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:45.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:45.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:45.385 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:22:45.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:22:45.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:45.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:46.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:47.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:47.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:47.183 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:22:47.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:22:47.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:47.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:47.922 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:48.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:48.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:48.924 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:22:48.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:22:48.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:48.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:49.690 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:50.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:50.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:50.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:22:50.692 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:22:50.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:50.693 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:51.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:52.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:52.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:52.565 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T09:22:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:22:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:52.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:22:53.054 INFO:tasks.workunit.client.0.smithi031.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1569536919,v1:127.0.0.1:6827/1569536919] [v2:127.0.0.1:6828/1569536919,v1:127.0.0.1:6829/1569536919] exists,up ec43e422-661b-4045-b207-f5bdeced86bd 2024-04-25T09:22:53.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:22:53.057 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:22:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:22:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:22:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-04-25T09:22:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:22:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:22:53.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:22:53.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:22:53.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:22:53.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-04-25T09:22:53.062 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:22:53.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b07c3b7-39d6-4de8-bb2d-ec0b82885aad 2024-04-25T09:22:53.062 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2b07c3b7-39d6-4de8-bb2d-ec0b82885aad' 2024-04-25T09:22:53.062 INFO:tasks.workunit.client.0.smithi031.stdout:add osd4 2b07c3b7-39d6-4de8-bb2d-ec0b82885aad 2024-04-25T09:22:53.062 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:22:53.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtICpm4IUtBhAAxOL9JGP8vKevVBruN08TUw== 2024-04-25T09:22:53.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtICpm4IUtBhAAxOL9JGP8vKevVBruN08TUw=="}' 2024-04-25T09:22:53.088 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b07c3b7-39d6-4de8-bb2d-ec0b82885aad -i td/repeer-on-acting-back/4/new.json 2024-04-25T09:22:53.592 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:22:53.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-04-25T09:22:53.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtICpm4IUtBhAAxOL9JGP8vKevVBruN08TUw== --osd-uuid 2b07c3b7-39d6-4de8-bb2d-ec0b82885aad 2024-04-25T09:22:53.651 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.662+0000 7fd7292027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:53.653 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.666+0000 7fd7292027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:53.654 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.666+0000 7fd7292027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:53.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.666+0000 7fd7292027c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:22:53.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.666+0000 7fd7292027c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:22:53.655 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.666+0000 7fd7292027c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:22:53.693 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:22:53.706+0000 7fd7292027c0 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-04-25T09:22:59.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-04-25T09:22:59.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:22:59.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T09:22:59.561 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd4 key to auth repository 2024-04-25T09:22:59.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:23:00.160 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.4 2024-04-25T09:23:00.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T09:23:00.160 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:23:00.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:23:00.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:23:00.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:23:00.201 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:00.214+0000 7fa18cdcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:00.203 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:00.214+0000 7fa18cdcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:00.204 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:00.218+0000 7fa18cdcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:00.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T09:23:00.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:23:00.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T09:23:00.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:23:00.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:23:00.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:00.631 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:23:00.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:23:00.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:00.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:00.753 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:00.766+0000 7fa18cdcb7c0 -1 Falling back to public interface 2024-04-25T09:23:01.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:02.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:02.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:02.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:23:02.114 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:23:02.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:02.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:02.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:03.283 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:03.294+0000 7fa18cdcb7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T09:23:03.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:03.592 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:23:03.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:03.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:23:03.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:03.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:04.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:05.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:05.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:05.184 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:23:05.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:23:05.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:05.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:05.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:06.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:06.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:06.646 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:23:06.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:23:06.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:06.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:07.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:08.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:08.198 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:08.198 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:23:08.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:23:08.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:08.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:08.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:09.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:09.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:09.657 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:23:09.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:23:09.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:09.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:10.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:11.341 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:11.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:11.342 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:23:11.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:23:11.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:11.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:11.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:12.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:23:12.791 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:23:12.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:12.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:13.248 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:14.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:14.250 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:14.250 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:23:14.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:23:14.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:14.251 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:14.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:15.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:15.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:15.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:23:15.898 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:23:15.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:15.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:16.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:17.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:17.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:17.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:23:17.363 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:23:17.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:17.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:17.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:18.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:18.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:18.912 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:23:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:23:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:18.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:19.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:20.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:20.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:20.778 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:23:20.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:23:20.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:20.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:21.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:22.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:22.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:22.768 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:23:22.768 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:23:22.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:22.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:23.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:24.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:24.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:24.556 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:23:24.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:23:24.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:24.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:25.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:26.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:26.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:26.495 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:23:26.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:23:26.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:26.495 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:27.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:28.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:28.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:28.295 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T09:23:28.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:23:28.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:28.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:23:28.793 INFO:tasks.workunit.client.0.smithi031.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/405323263,v1:127.0.0.1:6835/405323263] [v2:127.0.0.1:6836/405323263,v1:127.0.0.1:6837/405323263] exists,up 2b07c3b7-39d6-4de8-bb2d-ec0b82885aad 2024-04-25T09:23:28.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:23:28.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:23:28.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:23:28.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:23:28.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-04-25T09:23:28.801 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:23:28.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca4c0489-048d-4cd1-8e61-835250adf6c6 2024-04-25T09:23:28.801 INFO:tasks.workunit.client.0.smithi031.stdout:add osd5 ca4c0489-048d-4cd1-8e61-835250adf6c6 2024-04-25T09:23:28.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ca4c0489-048d-4cd1-8e61-835250adf6c6' 2024-04-25T09:23:28.802 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:23:28.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQISpmvi9yMhAALxxvoZGH1vDyb/gjQoDKig== 2024-04-25T09:23:28.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQISpmvi9yMhAALxxvoZGH1vDyb/gjQoDKig=="}' 2024-04-25T09:23:28.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca4c0489-048d-4cd1-8e61-835250adf6c6 -i td/repeer-on-acting-back/5/new.json 2024-04-25T09:23:29.334 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:23:29.351 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-04-25T09:23:29.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQISpmvi9yMhAALxxvoZGH1vDyb/gjQoDKig== --osd-uuid ca4c0489-048d-4cd1-8e61-835250adf6c6 2024-04-25T09:23:29.396 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.410+0000 7f854ccc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:29.398 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.410+0000 7f854ccc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:29.399 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.414+0000 7f854ccc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:29.400 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.414+0000 7f854ccc67c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:23:29.400 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.414+0000 7f854ccc67c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:23:29.400 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.414+0000 7f854ccc67c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:23:29.443 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:29.454+0000 7f854ccc67c0 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-04-25T09:23:35.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-04-25T09:23:35.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:23:35.694 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd5 key to auth repository 2024-04-25T09:23:35.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T09:23:35.694 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:23:36.336 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.5 2024-04-25T09:23:36.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T09:23:36.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:23:36.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:23:36.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:23:36.338 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:23:36.377 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:36.390+0000 7fed44cfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:36.401 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:36.414+0000 7fed44cfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:36.402 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:36.414+0000 7fed44cfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:36.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T09:23:36.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:23:36.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T09:23:36.842 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:23:36.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:23:36.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:23:36.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:36.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:23:36.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:36.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:36.960 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:36.974+0000 7fed44cfd7c0 -1 Falling back to public interface 2024-04-25T09:23:37.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:38.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:38.326 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:38.327 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:23:38.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:23:38.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:38.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:38.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:39.469 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:23:39.482+0000 7fed44cfd7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T09:23:39.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:39.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:39.801 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:23:39.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:23:39.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:39.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:40.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:41.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:41.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:41.295 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:23:41.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:23:41.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:41.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:41.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:42.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:42.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:42.746 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:23:42.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:23:42.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:42.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:43.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:44.307 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:23:44.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:23:44.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:44.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:44.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:45.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:45.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:45.776 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:23:45.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:23:45.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:45.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:46.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:47.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:47.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:47.329 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:23:47.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:23:47.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:47.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:47.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:48.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:48.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:48.776 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:23:48.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:23:48.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:48.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:49.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:50.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:50.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:50.316 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:23:50.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:23:50.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:50.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:50.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:51.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:51.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:51.891 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:23:51.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:23:51.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:51.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:52.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:53.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:53.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:53.508 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:23:53.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:23:53.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:53.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:54.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:55.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:55.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:55.597 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:23:55.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:23:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:55.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:56.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:57.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:57.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:57.299 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:23:57.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:23:57.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:57.299 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:23:58.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:59.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:59.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:59.012 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:23:59.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:23:59.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:59.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:24:00.051 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:01.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:01.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:01.053 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:24:01.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:24:01.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:01.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:24:01.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:02.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:02.814 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:02.814 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:24:02.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:24:02.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:02.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:24:03.317 INFO:tasks.workunit.client.0.smithi031.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2452462502,v1:127.0.0.1:6843/2452462502] [v2:127.0.0.1:6844/2452462502,v1:127.0.0.1:6845/2452462502] exists,up ca4c0489-048d-4cd1-8e61-835250adf6c6 2024-04-25T09:24:03.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:03.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:03.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:03.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-04-25T09:24:03.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:24:03.912 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T09:24:03.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:24:04.931 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-04-25T09:24:05.785 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 size to 3 2024-04-25T09:24:05.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-04-25T09:24:06.622 INFO:tasks.workunit.client.0.smithi031.stderr:set pool 1 min_size to 2 2024-04-25T09:24:06.647 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-04-25T09:24:06.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-04-25T09:24:06.648 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-04-25T09:24:07.076 INFO:tasks.workunit.client.0.smithi031.stderr:dumped pools 2024-04-25T09:24:07.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-04-25T09:24:07.092 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-04-25T09:24:07.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-04-25T09:24:07.727 INFO:tasks.workunit.client.0.smithi031.stderr:set require_min_compat_client to luminous 2024-04-25T09:24:07.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-04-25T09:24:08.541 INFO:tasks.workunit.client.0.smithi031.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-04-25T09:24:08.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-04-25T09:24:08.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:08.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:09.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:24:09.090 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:24:09.090 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:24:09.090 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T09:24:09.091 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T09:24:09.091 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T09:24:09.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:09.091 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:09.091 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:09.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836522 2024-04-25T09:24:09.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836522 2024-04-25T09:24:09.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522' 2024-04-25T09:24:09.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:09.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:24:09.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672995 2024-04-25T09:24:09.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672995 2024-04-25T09:24:09.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672995' 2024-04-25T09:24:09.427 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:09.427 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:24:09.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509472 2024-04-25T09:24:09.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509472 2024-04-25T09:24:09.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672995 2-64424509472' 2024-04-25T09:24:09.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:09.595 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:24:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345939 2024-04-25T09:24:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345939 2024-04-25T09:24:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672995 2-64424509472 3-85899345939' 2024-04-25T09:24:09.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:09.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:24:09.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182420 2024-04-25T09:24:09.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182420 2024-04-25T09:24:09.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672995 2-64424509472 3-85899345939 4-107374182420' 2024-04-25T09:24:09.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:09.927 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:24:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018886 2024-04-25T09:24:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018886 2024-04-25T09:24:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836522 1-42949672995 2-64424509472 3-85899345939 4-107374182420 5-128849018886' 2024-04-25T09:24:10.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:10.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836522 2024-04-25T09:24:10.099 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:10.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:10.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836522 2024-04-25T09:24:10.101 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:10.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836522 2024-04-25T09:24:10.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-04-25T09:24:10.102 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836522 2024-04-25T09:24:10.102 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:10.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836522 -lt 21474836522 2024-04-25T09:24:10.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:10.537 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672995 2024-04-25T09:24:10.538 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:10.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:24:10.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672995 2024-04-25T09:24:10.540 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:10.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672995 2024-04-25T09:24:10.542 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672995 2024-04-25T09:24:10.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-04-25T09:24:10.542 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:24:11.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672995 -lt 42949672995 2024-04-25T09:24:11.071 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:11.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509472 2024-04-25T09:24:11.071 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:11.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:24:11.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509472 2024-04-25T09:24:11.073 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:11.074 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509472 2024-04-25T09:24:11.075 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509472 2024-04-25T09:24:11.075 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-04-25T09:24:11.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:24:11.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509472 -lt 64424509472 2024-04-25T09:24:11.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:11.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345939 2024-04-25T09:24:11.558 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:11.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:24:11.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:11.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345939 2024-04-25T09:24:11.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345939 2024-04-25T09:24:11.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-04-25T09:24:11.561 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345939 2024-04-25T09:24:11.561 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:24:12.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345939 -lt 85899345939 2024-04-25T09:24:12.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:12.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182420 2024-04-25T09:24:12.003 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:12.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:24:12.006 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182420 2024-04-25T09:24:12.006 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:12.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182420 2024-04-25T09:24:12.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182420' 2024-04-25T09:24:12.007 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182420 2024-04-25T09:24:12.008 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:24:12.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182420 -lt 107374182420 2024-04-25T09:24:12.484 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:12.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018886 2024-04-25T09:24:12.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:12.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:24:12.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018886 2024-04-25T09:24:12.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:12.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018886 2024-04-25T09:24:12.488 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018886 2024-04-25T09:24:12.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-04-25T09:24:12.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:24:12.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018886 -lt 128849018886 2024-04-25T09:24:12.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:24:12.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:12.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:12.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:12.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:24:12.960 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:12.960 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:12.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:12.960 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:12.961 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:13.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:24:13.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:13.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:13.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:13.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:13.063 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:13.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:13.527 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:13.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836524 2024-04-25T09:24:13.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836524 2024-04-25T09:24:13.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524' 2024-04-25T09:24:13.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:13.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:24:13.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-04-25T09:24:13.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-04-25T09:24:13.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998' 2024-04-25T09:24:13.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:13.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:24:14.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509475 2024-04-25T09:24:14.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509475 2024-04-25T09:24:14.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509475' 2024-04-25T09:24:14.038 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:14.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:24:14.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T09:24:14.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T09:24:14.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509475 3-85899345942' 2024-04-25T09:24:14.209 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:14.209 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:24:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182423 2024-04-25T09:24:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182423 2024-04-25T09:24:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509475 3-85899345942 4-107374182423' 2024-04-25T09:24:14.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:14.364 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:24:14.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018889 2024-04-25T09:24:14.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018889 2024-04-25T09:24:14.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509475 3-85899345942 4-107374182423 5-128849018889' 2024-04-25T09:24:14.518 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:14.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:14.519 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836524 2024-04-25T09:24:14.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:14.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836524 2024-04-25T09:24:14.521 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:14.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836524 2024-04-25T09:24:14.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-04-25T09:24:14.522 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836524 2024-04-25T09:24:14.522 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:14.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836524 -lt 21474836524 2024-04-25T09:24:14.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:14.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-04-25T09:24:14.944 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:14.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:24:14.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-04-25T09:24:14.947 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:14.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-04-25T09:24:14.948 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949672998 2024-04-25T09:24:14.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-04-25T09:24:14.948 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:24:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672996 -lt 42949672998 2024-04-25T09:24:15.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:24:16.492 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:24:16.493 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:24:16.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-04-25T09:24:16.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:16.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509475 2024-04-25T09:24:16.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:16.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:24:16.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509475 2024-04-25T09:24:16.941 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:16.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509475 2024-04-25T09:24:16.943 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509475 2024-04-25T09:24:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-04-25T09:24:16.943 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:24:17.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509475 -lt 64424509475 2024-04-25T09:24:17.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:17.440 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345942 2024-04-25T09:24:17.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:17.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:24:17.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345942 2024-04-25T09:24:17.444 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:17.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T09:24:17.445 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345942 2024-04-25T09:24:17.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-04-25T09:24:17.446 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:24:17.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345943 -lt 85899345942 2024-04-25T09:24:17.898 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:17.898 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182423 2024-04-25T09:24:17.899 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:17.900 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:24:17.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182423 2024-04-25T09:24:17.901 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:17.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182423 2024-04-25T09:24:17.903 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182423 2024-04-25T09:24:17.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182423' 2024-04-25T09:24:17.903 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:24:18.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182423 -lt 107374182423 2024-04-25T09:24:18.371 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:18.372 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018889 2024-04-25T09:24:18.373 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:18.374 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:24:18.375 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018889 2024-04-25T09:24:18.376 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:18.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018889 2024-04-25T09:24:18.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2024-04-25T09:24:18.377 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018889 2024-04-25T09:24:18.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:24:18.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018889 -lt 128849018889 2024-04-25T09:24:18.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:24:18.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:18.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:19.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:24:19.440 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:19.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:19.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:19.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:19.441 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:19.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:19.442 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:19.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:24:19.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:19.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:19.857 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:20.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:24:20.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:24:20.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:24:20.423 INFO:tasks.workunit.client.0.smithi031.stdout:writing initial objects 2024-04-25T09:24:20.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-04-25T09:24:20.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-04-25T09:24:20.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:20.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-04-25T09:24:20.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:20.609 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-04-25T09:24:20.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:20.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-04-25T09:24:20.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:20.919 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-04-25T09:24:21.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.070 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-04-25T09:24:21.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-04-25T09:24:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.363 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-04-25T09:24:21.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-04-25T09:24:21.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-04-25T09:24:21.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-04-25T09:24:21.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:21.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-04-25T09:24:22.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:22.078 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-04-25T09:24:22.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:22.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-04-25T09:24:22.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:22.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-04-25T09:24:22.770 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:22.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-04-25T09:24:22.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:22.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-04-25T09:24:23.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:23.080 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-04-25T09:24:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:23.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-04-25T09:24:23.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:23.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-04-25T09:24:23.532 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:23.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-04-25T09:24:23.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:23.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-04-25T09:24:23.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:23.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-04-25T09:24:24.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-04-25T09:24:24.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-04-25T09:24:24.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-04-25T09:24:24.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-04-25T09:24:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-04-25T09:24:24.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-04-25T09:24:24.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:24.933 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-04-25T09:24:25.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:25.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-04-25T09:24:25.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:25.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-04-25T09:24:25.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:25.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-04-25T09:24:25.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:25.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-04-25T09:24:25.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:25.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-04-25T09:24:25.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:25.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-04-25T09:24:26.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:26.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-04-25T09:24:26.231 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:26.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-04-25T09:24:26.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:26.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-04-25T09:24:26.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:26.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-04-25T09:24:26.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:26.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-04-25T09:24:26.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:26.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-04-25T09:24:27.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.002 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-04-25T09:24:27.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-04-25T09:24:27.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-04-25T09:24:27.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.504 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-04-25T09:24:27.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-04-25T09:24:27.847 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-04-25T09:24:27.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:27.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-04-25T09:24:28.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:28.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-04-25T09:24:28.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:28.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-04-25T09:24:28.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:28.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-04-25T09:24:28.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:28.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-04-25T09:24:28.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:28.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-04-25T09:24:28.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:28.954 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-04-25T09:24:29.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:29.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-04-25T09:24:29.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:29.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-04-25T09:24:29.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:29.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-04-25T09:24:29.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:29.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-04-25T09:24:29.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:29.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-04-25T09:24:29.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:29.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-04-25T09:24:30.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-04-25T09:24:30.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.227 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-04-25T09:24:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-04-25T09:24:30.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.546 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-04-25T09:24:30.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-04-25T09:24:30.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-04-25T09:24:30.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:30.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-04-25T09:24:31.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:31.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-04-25T09:24:31.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:31.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-04-25T09:24:31.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:31.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-04-25T09:24:31.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:31.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-04-25T09:24:31.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:31.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-04-25T09:24:32.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-04-25T09:24:32.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-04-25T09:24:32.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-04-25T09:24:32.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-04-25T09:24:32.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.649 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-04-25T09:24:32.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.808 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-04-25T09:24:32.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:32.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-04-25T09:24:33.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:33.144 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-04-25T09:24:33.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:33.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-04-25T09:24:33.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:33.453 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-04-25T09:24:33.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:33.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-04-25T09:24:33.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:33.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-04-25T09:24:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:33.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-04-25T09:24:34.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-04-25T09:24:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-04-25T09:24:34.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-04-25T09:24:34.525 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.526 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-04-25T09:24:34.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-04-25T09:24:34.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.821 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-04-25T09:24:34.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:34.980 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-04-25T09:24:35.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:35.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-04-25T09:24:35.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:35.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-04-25T09:24:35.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:35.441 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-04-25T09:24:35.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:35.592 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-04-25T09:24:35.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:35.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-04-25T09:24:35.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:35.918 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-04-25T09:24:36.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:36.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-04-25T09:24:36.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:24:36.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-04-25T09:24:36.380 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:24:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:24:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:36.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:36.381 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:24:36.382 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:36.382 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:36.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:36.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:36.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:36.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:24:36.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:36.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:36.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:36.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:36.459 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:36.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:24:36.939 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:36.940 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836531 2024-04-25T09:24:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836531 2024-04-25T09:24:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531' 2024-04-25T09:24:37.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:37.116 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:24:37.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673004 2024-04-25T09:24:37.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673004 2024-04-25T09:24:37.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673004' 2024-04-25T09:24:37.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:37.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:24:37.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509481 2024-04-25T09:24:37.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509481 2024-04-25T09:24:37.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673004 2-64424509481' 2024-04-25T09:24:37.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:37.443 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:24:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345949 2024-04-25T09:24:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345949 2024-04-25T09:24:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673004 2-64424509481 3-85899345949' 2024-04-25T09:24:37.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:37.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:24:37.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182429 2024-04-25T09:24:37.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182429 2024-04-25T09:24:37.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673004 2-64424509481 3-85899345949 4-107374182429' 2024-04-25T09:24:37.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:37.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:24:37.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018895 2024-04-25T09:24:37.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018895 2024-04-25T09:24:37.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673004 2-64424509481 3-85899345949 4-107374182429 5-128849018895' 2024-04-25T09:24:37.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:37.935 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836531 2024-04-25T09:24:37.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:37.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:37.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836531 2024-04-25T09:24:37.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:37.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836531 2024-04-25T09:24:37.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-04-25T09:24:37.938 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836531 2024-04-25T09:24:37.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836531 -lt 21474836531 2024-04-25T09:24:38.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:38.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673004 2024-04-25T09:24:38.401 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:38.402 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:24:38.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673004 2024-04-25T09:24:38.403 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:38.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673004 2024-04-25T09:24:38.404 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673004 2024-04-25T09:24:38.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-04-25T09:24:38.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:24:38.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673005 -lt 42949673004 2024-04-25T09:24:38.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:38.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509481 2024-04-25T09:24:38.894 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:38.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:24:38.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509481 2024-04-25T09:24:38.896 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:38.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509481 2024-04-25T09:24:38.897 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-04-25T09:24:38.897 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509481 2024-04-25T09:24:38.897 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:24:39.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509481 -lt 64424509481 2024-04-25T09:24:39.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:39.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:39.334 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345949 2024-04-25T09:24:39.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:24:39.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345949 2024-04-25T09:24:39.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:39.336 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345949 2024-04-25T09:24:39.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2024-04-25T09:24:39.337 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345949 2024-04-25T09:24:39.337 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:24:39.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345949 -lt 85899345949 2024-04-25T09:24:39.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:39.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:39.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182429 2024-04-25T09:24:39.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:24:39.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182429 2024-04-25T09:24:39.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:39.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182429 2024-04-25T09:24:39.775 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182429 2024-04-25T09:24:39.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182429' 2024-04-25T09:24:39.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:24:40.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182430 -lt 107374182429 2024-04-25T09:24:40.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:40.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018895 2024-04-25T09:24:40.218 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:40.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:24:40.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018895 2024-04-25T09:24:40.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:40.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018895 2024-04-25T09:24:40.222 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018895 2024-04-25T09:24:40.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018895' 2024-04-25T09:24:40.222 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:24:40.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018896 -lt 128849018895 2024-04-25T09:24:40.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:24:40.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:40.690 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:41.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:24:41.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:41.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:41.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:41.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:41.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:41.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:41.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:41.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:24:41.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:41.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:41.681 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:42.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:24:42.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:24:42.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:24:42.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-04-25T09:24:43.067 INFO:tasks.workunit.client.0.smithi031.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-04-25T09:24:43.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:24:45.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:24:45.089 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:24:45.090 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:24:45.279 INFO:tasks.workunit.client.0.smithi031.stdout: 2, 2024-04-25T09:24:45.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-04-25T09:24:45.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:24:45.280 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:24:45.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:24:45.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:24:45.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:24:45.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:24:45.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-04-25T09:24:46.158 INFO:tasks.workunit.client.0.smithi031.stderr:osd.2 is already down. 2024-04-25T09:24:46.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:24:48.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:24:48.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:24:48.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:48.421 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:24:48.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:24:48.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:24:48.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:24:48.423 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=02070f58-2a12-4f05-943b-7f1f442f4b83 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.470537/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:24:48.424 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:24:48.461 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:24:48.472+0000 7f17ba3e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:48.463 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:24:48.476+0000 7f17ba3e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:48.464 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:24:48.476+0000 7f17ba3e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:48.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:24:48.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:24:48.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:24:48.867 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:24:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:24:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:24:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:24:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:48.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:49.036 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:24:49.048+0000 7f17ba3e27c0 -1 Falling back to public interface 2024-04-25T09:24:49.375 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:50.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:50.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:50.377 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:24:50.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:24:50.377 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:50.378 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:50.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:51.829 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:51.830 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:24:51.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:24:51.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:51.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:52.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:52.945 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:24:52.956+0000 7f17ba3e27c0 -1 osd.2 46 log_to_monitors true 2024-04-25T09:24:53.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:53.330 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:53.330 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:24:53.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:24:53.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:53.331 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:53.988 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/2337780271,v1:127.0.0.1:6819/2337780271] [v2:127.0.0.1:6820/2337780271,v1:127.0.0.1:6821/2337780271] exists,up d79f2943-4567-402a-ae88-1731af2bad2d 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:24:53.990 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:24:53.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:24:53.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:53.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:24:53.991 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:24:53.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:53.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:54.745 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/2337780271,v1:127.0.0.1:6819/2337780271] [v2:127.0.0.1:6820/2337780271,v1:127.0.0.1:6821/2337780271] exists,up d79f2943-4567-402a-ae88-1731af2bad2d 2024-04-25T09:24:54.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:54.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:54.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:24:56.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:24:56.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:24:56.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:24:56.966 INFO:tasks.workunit.client.0.smithi031.stdout: 2 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:56.967 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:56.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:56.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:56.968 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:57.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:24:57.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:57.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:57.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:57.036 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:57.036 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:2 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:3 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:4 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:5' 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:57.480 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:57.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836537 2024-04-25T09:24:57.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836537 2024-04-25T09:24:57.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537' 2024-04-25T09:24:57.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:57.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:24:57.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673011 2024-04-25T09:24:57.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673011 2024-04-25T09:24:57.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011' 2024-04-25T09:24:57.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:57.978 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:24:58.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364803 2024-04-25T09:24:58.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364803 2024-04-25T09:24:58.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-214748364803' 2024-04-25T09:24:58.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:58.217 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:24:58.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345955 2024-04-25T09:24:58.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345955 2024-04-25T09:24:58.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-214748364803 3-85899345955' 2024-04-25T09:24:58.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:58.487 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:24:58.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182436 2024-04-25T09:24:58.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182436 2024-04-25T09:24:58.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-214748364803 3-85899345955 4-107374182436' 2024-04-25T09:24:58.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:58.662 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:24:58.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018901 2024-04-25T09:24:58.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018901 2024-04-25T09:24:58.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-214748364803 3-85899345955 4-107374182436 5-128849018901' 2024-04-25T09:24:58.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:58.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836537 2024-04-25T09:24:58.831 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:58.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:58.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836537 2024-04-25T09:24:58.833 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:58.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836537 2024-04-25T09:24:58.834 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836537 2024-04-25T09:24:58.835 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-04-25T09:24:58.835 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:59.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836537 -lt 21474836537 2024-04-25T09:24:59.263 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:59.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673011 2024-04-25T09:24:59.263 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:59.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:24:59.265 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673011 2024-04-25T09:24:59.266 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:59.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673011 2024-04-25T09:24:59.267 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 42949673011 2024-04-25T09:24:59.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-04-25T09:24:59.267 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:24:59.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673009 -lt 42949673011 2024-04-25T09:24:59.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:25:00.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:25:00.757 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:25:01.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673011 -lt 42949673011 2024-04-25T09:25:01.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:01.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364803 2024-04-25T09:25:01.211 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:01.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:25:01.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364803 2024-04-25T09:25:01.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:01.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364803 2024-04-25T09:25:01.215 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 214748364803 2024-04-25T09:25:01.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-04-25T09:25:01.216 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:25:01.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364804 -lt 214748364803 2024-04-25T09:25:01.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:01.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345955 2024-04-25T09:25:01.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:01.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:25:01.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345955 2024-04-25T09:25:01.778 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:01.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345955 2024-04-25T09:25:01.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2024-04-25T09:25:01.779 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.3 seq 85899345955 2024-04-25T09:25:01.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:25:02.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345955 -lt 85899345955 2024-04-25T09:25:02.421 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:02.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182436 2024-04-25T09:25:02.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:02.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:25:02.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182436 2024-04-25T09:25:02.424 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:02.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182436 2024-04-25T09:25:02.425 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.4 seq 107374182436 2024-04-25T09:25:02.425 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182436' 2024-04-25T09:25:02.426 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:25:03.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182436 -lt 107374182436 2024-04-25T09:25:03.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:03.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018901 2024-04-25T09:25:03.020 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:03.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:25:03.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018901 2024-04-25T09:25:03.023 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:03.024 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018901 2024-04-25T09:25:03.025 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.5 seq 128849018901 2024-04-25T09:25:03.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-04-25T09:25:03.025 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:25:03.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018902 -lt 128849018901 2024-04-25T09:25:03.562 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:25:03.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:03.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:04.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:25:04.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:25:04.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:25:04.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:25:04.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:25:04.329 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:25:04.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:25:04.330 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:25:04.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:25:04.764 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:25:04.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:04.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:25:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:25:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:25:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:25:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:25:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:25:05.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:25:05.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:25:05.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:25:05.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:25:05.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:25:05.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:25:05.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:25:05.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:25:05.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:25:05.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:25:05.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:05.865 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:06.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:25:06.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:25:06.437 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:25:06.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:25:06.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:25:06.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:25:06.438 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:25:06.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:25:06.438 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:25:07.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2470 2024-04-25T09:25:07.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2470 '!=' null 2024-04-25T09:25:07.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:25:07.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:25:07.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:25:07.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:25:07.128 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:25:07.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:25:07.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:25:07.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:25:07.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:25:07.129 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:25:07.130 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:25:07.548 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:25:07.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:25:07.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:07.549 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:25:08.110 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:25:08.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2470 2024-04-25T09:25:08.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2470 '!=' null 2024-04-25T09:25:08.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:25:08.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:25:08.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:25:08.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:25:08.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:25:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:25:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:25:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:25:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:25:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:25:08.777 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:25:09.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:25:09.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:25:09.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:09.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:09.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:25:09.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:25:09.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:25:09.765 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-04-25T09:25:09.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.0.log 2024-04-25T09:25:09.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-04-25T09:25:09.776 INFO:tasks.workunit.client.0.smithi031.stdout:success 2024-04-25T09:25:09.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-04-25T09:25:09.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:25:09.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:25:10.376 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' does not exist 2024-04-25T09:25:10.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-04-25T09:25:10.395 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:25:10.395 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:25:10.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:25:10.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:25:10.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:25:15.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:25:15.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:25:15.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:25:15.724 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:25:15.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:25:15.725 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:25:15.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:25:15.726 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:25:15.726 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:25:15.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:25:15.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:25:15.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:25:15.728 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:25:15.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:25:15.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:25:15.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:25:15.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:25:15.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:15.769 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:25:15.769 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470537 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repeer-on-acting-back 0 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:25:15.771 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:25:15.772 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:25:15.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:25:15.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:25:15.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:25:15.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:25:15.773 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:25:15.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:25:15.774 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:25:15.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:25:15.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:25:15.775 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:25:15.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:25:15.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:25:15.776 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:25:15.776 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:25:15.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:25:15.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:25:15.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:25:15.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:25:15.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:15.779 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.470537 2024-04-25T09:25:15.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.470537 2024-04-25T09:25:15.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:25:15.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:25:15.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:25:15.781 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:25:15.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:25:15.793 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-04-25T09:25:15.793 DEBUG:teuthology.orchestra.run.smithi031:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-04-25T09:25:15.848 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:25:15.853 INFO:tasks.workunit.client.0.smithi031.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:25:15.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repro-long-log 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-04-25T09:25:15.854 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-04-25T09:25:15.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-04-25T09:25:15.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none ' 2024-04-25T09:25:15.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T09:25:15.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-04-25T09:25:15.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_repro_long_log2 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_trim_max_entries 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:TEST_trim_max_entries_with_dups' 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:25:15.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:25:15.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:25:15.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:25:15.859 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:25:15.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:25:15.860 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:25:15.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:25:15.861 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:25:15.861 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:25:15.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:25:15.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:25:15.862 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:25:15.862 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:25:15.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:25:15.864 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:25:15.865 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:25:15.866 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:25:15.866 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:15.866 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:15.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:25:15.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:25:15.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:25:15.867 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:25:15.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:25:15.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:15.868 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:15.868 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.486059 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-04-25T09:25:15.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:25:15.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:25:16.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:25:16.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:25:16.335 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:25:16.336 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:25:16.336 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:16.336 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:16.336 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:25:16.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:25:16.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:25:16.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:25:16.556 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:25:16.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:25:16.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:25:16.557 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:25:16.557 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:25:16.557 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:25:16.558 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:25:16.558 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:25:16.558 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:16.559 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:16.559 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:25:16.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:25:16.559 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get fsid 2024-04-25T09:25:16.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:25:16.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:25:16.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:25:16.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:25:16.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:25:16.679 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:25:16.679 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:25:16.679 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:25:16.679 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:25:16.680 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:16.680 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:16.680 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:25:16.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:25:16.680 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get mon_host 2024-04-25T09:25:16.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:25:16.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:25:16.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:25:16.790 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:25:16.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:25:16.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:25:16.791 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:25:17.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:25:17.703 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:25:17.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:25:17.704 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:25:17.704 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:17.704 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:17.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:25:17.704 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:25:17.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:25:17.745 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:17.746 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:25:17.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:25:17.748 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:25:17.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5aeeb25-8bf8-4deb-ab7b-c8ff0819cf32 2024-04-25T09:25:17.748 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d5aeeb25-8bf8-4deb-ab7b-c8ff0819cf32' 2024-04-25T09:25:17.748 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 d5aeeb25-8bf8-4deb-ab7b-c8ff0819cf32 2024-04-25T09:25:17.749 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:25:17.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9ISpmIsWELhAAF1n3xwGdWrbz7wyerCpGoQ== 2024-04-25T09:25:17.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9ISpmIsWELhAAF1n3xwGdWrbz7wyerCpGoQ=="}' 2024-04-25T09:25:17.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5aeeb25-8bf8-4deb-ab7b-c8ff0819cf32 -i td/repro-long-log/0/new.json 2024-04-25T09:25:18.067 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:25:18.085 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:25:18.086 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9ISpmIsWELhAAF1n3xwGdWrbz7wyerCpGoQ== --osd-uuid d5aeeb25-8bf8-4deb-ab7b-c8ff0819cf32 2024-04-25T09:25:18.120 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.132+0000 7f5dbf6887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:18.135 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.148+0000 7f5dbf6887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:18.136 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.148+0000 7f5dbf6887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:18.137 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.148+0000 7f5dbf6887c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:25:18.137 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.148+0000 7f5dbf6887c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:25:18.137 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.148+0000 7f5dbf6887c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:25:18.175 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:18.188+0000 7f5dbf6887c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:25:23.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:25:23.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:25:23.983 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:25:23.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:25:23.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:25:24.543 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:25:24.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:25:24.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:25:24.574 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:24.584+0000 7fada4a8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:24.632 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:24.644+0000 7fada4a8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:24.634 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:24.644+0000 7fada4a8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:24.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:25:24.960 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:25:24.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:24.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:25.189 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:25.200+0000 7fada4a8e7c0 -1 Falling back to public interface 2024-04-25T09:25:25.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:26.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:26.366 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:26.367 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:25:26.367 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:25:26.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:26.368 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:26.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:27.695 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:27.708+0000 7fada4a8e7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:25:27.840 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:27.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:27.841 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:25:27.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:25:27.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:27.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:28.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:29.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:29.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:29.308 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:25:29.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:25:29.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:29.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:29.736 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:30.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:30.737 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:30.738 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:25:30.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:25:30.738 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:30.739 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:31.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:32.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:32.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:32.180 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:25:32.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:25:32.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:32.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:32.636 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:33.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:33.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:33.638 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:25:33.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:25:33.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:33.639 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:34.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:35.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:35.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:35.115 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:25:35.115 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:25:35.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:35.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:36.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:36.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:36.566 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:25:36.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:25:36.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:36.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:37.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:38.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:38.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:38.005 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:25:38.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:25:38.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:38.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:38.521 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:39.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:39.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:39.523 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:25:39.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:25:39.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:39.524 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:40.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:41.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:41.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:41.305 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:25:41.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:25:41.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:41.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:41.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:42.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:42.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:42.946 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:25:42.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:25:42.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:42.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:43.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:44.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:44.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:44.815 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:25:44.816 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:25:44.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:44.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:45.912 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:46.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:46.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:46.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:25:46.915 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:25:46.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:46.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:47.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:48.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:48.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:48.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:25:48.958 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:25:48.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:48.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:49.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:50.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:50.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:50.508 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:25:50.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:25:50.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:50.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:51.006 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/870658387,v1:127.0.0.1:6803/870658387] [v2:127.0.0.1:6804/870658387,v1:127.0.0.1:6805/870658387] exists,up d5aeeb25-8bf8-4deb-ab7b-c8ff0819cf32 2024-04-25T09:25:51.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:25:51.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:25:51.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:25:51.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:25:51.012 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:25:51.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28e37f60-3ee1-44c9-a34f-9170efcd4381 2024-04-25T09:25:51.013 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 28e37f60-3ee1-44c9-a34f-9170efcd4381' 2024-04-25T09:25:51.013 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 28e37f60-3ee1-44c9-a34f-9170efcd4381 2024-04-25T09:25:51.013 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:25:51.025 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfISpm6Vh/AhAAEZeVZccmAjO+Co/fPjEQ9A== 2024-04-25T09:25:51.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfISpm6Vh/AhAAEZeVZccmAjO+Co/fPjEQ9A=="}' 2024-04-25T09:25:51.026 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28e37f60-3ee1-44c9-a34f-9170efcd4381 -i td/repro-long-log/1/new.json 2024-04-25T09:25:51.514 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:25:51.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:25:51.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfISpm6Vh/AhAAEZeVZccmAjO+Co/fPjEQ9A== --osd-uuid 28e37f60-3ee1-44c9-a34f-9170efcd4381 2024-04-25T09:25:51.557 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.571+0000 7fe8561e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:51.561 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.575+0000 7fe8561e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:51.562 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.575+0000 7fe8561e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:51.563 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.575+0000 7fe8561e97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:25:51.563 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.575+0000 7fe8561e97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:25:51.563 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.575+0000 7fe8561e97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:25:51.600 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:51.615+0000 7fe8561e97c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:25:57.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:25:57.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:25:57.766 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:25:57.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:25:57.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:25:58.346 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:25:58.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:25:58.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:25:58.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:25:58.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:25:58.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:25:58.378 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:58.391+0000 7fe8e4dad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:58.396 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:58.411+0000 7fe8e4dad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:58.397 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:58.411+0000 7fe8e4dad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:58.809 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:25:58.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:25:58.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:58.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:58.952 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:25:58.967+0000 7fe8e4dad7c0 -1 Falling back to public interface 2024-04-25T09:25:59.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:00.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:00.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:00.305 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:26:00.305 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:26:00.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:00.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:00.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:01.453 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:01.467+0000 7fe8e4dad7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:26:01.778 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:01.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:01.779 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:26:01.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:26:01.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:01.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:02.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:03.241 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:03.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:03.242 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:26:03.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:26:03.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:03.243 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:03.831 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:04.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:04.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:04.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:26:04.834 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:26:04.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:04.834 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:05.269 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:06.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:06.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:06.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:26:06.272 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:26:06.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:06.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:06.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:07.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:07.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:07.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:26:07.708 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:26:07.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:07.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:08.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:09.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:09.285 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:09.286 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:26:09.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:26:09.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:09.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:09.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:10.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:10.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:10.723 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:26:10.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:26:10.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:10.724 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:11.230 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:12.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:12.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:12.232 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:26:12.232 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:26:12.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:12.233 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:12.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:13.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:13.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:13.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:26:13.670 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:26:13.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:13.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:14.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:15.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:15.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:15.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:26:15.256 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:26:15.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:15.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:16.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:17.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:17.302 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:17.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:26:17.303 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:26:17.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:17.304 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:18.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:19.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:19.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:19.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:26:19.169 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:26:19.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:19.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:19.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:20.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:20.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:20.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:26:20.929 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:26:20.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:20.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:21.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:22.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:22.687 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:22.687 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:26:22.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:26:22.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:22.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:24.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:25.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:25.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:25.171 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:26:25.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:26:25.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:25.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:25.647 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1047233860,v1:127.0.0.1:6811/1047233860] [v2:127.0.0.1:6812/1047233860,v1:127.0.0.1:6813/1047233860] exists,up 28e37f60-3ee1-44c9-a34f-9170efcd4381 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:26:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:26:25.652 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:26:25.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:26:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:26:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:26:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:26:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:26:25.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:26:25.654 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:26:25.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=354fb3c9-bd63-46fd-9500-b7bf7626d115 2024-04-25T09:26:25.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 354fb3c9-bd63-46fd-9500-b7bf7626d115' 2024-04-25T09:26:25.655 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 354fb3c9-bd63-46fd-9500-b7bf7626d115 2024-04-25T09:26:25.655 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:26:25.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBISpmxVj6KBAAwgll9qWfGpuzyrB0eyg26Q== 2024-04-25T09:26:25.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBISpmxVj6KBAAwgll9qWfGpuzyrB0eyg26Q=="}' 2024-04-25T09:26:25.671 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 354fb3c9-bd63-46fd-9500-b7bf7626d115 -i td/repro-long-log/2/new.json 2024-04-25T09:26:26.250 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:26:26.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:26:26.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBISpmxVj6KBAAwgll9qWfGpuzyrB0eyg26Q== --osd-uuid 354fb3c9-bd63-46fd-9500-b7bf7626d115 2024-04-25T09:26:26.306 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.319+0000 7fb7065b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:26.322 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.335+0000 7fb7065b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:26.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.339+0000 7fb7065b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:26.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.339+0000 7fb7065b07c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:26:26.325 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.339+0000 7fb7065b07c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:26:26.325 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.339+0000 7fb7065b07c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:26:26.431 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:26.447+0000 7fb7065b07c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:26:32.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:26:32.300 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:26:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:26:32.301 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:26:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:26:32.887 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:26:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:26:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:26:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:26:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:26:32.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:26:32.919 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:32.935+0000 7f6d4bebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:32.921 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:32.935+0000 7f6d4bebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:32.922 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:32.935+0000 7f6d4bebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:33.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:26:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:26:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:26:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:26:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:26:33.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:33.384 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:26:33.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:26:33.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:33.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:33.473 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:33.487+0000 7f6d4bebd7c0 -1 Falling back to public interface 2024-04-25T09:26:33.857 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:34.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:34.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:34.859 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:26:34.859 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:26:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:34.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:35.278 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:35.985 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:26:35.999+0000 7f6d4bebd7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:26:36.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:36.280 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:26:36.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:36.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:26:36.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:36.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:36.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:37.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:37.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:37.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:26:37.807 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:26:37.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:37.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:38.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:39.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:39.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:39.255 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:26:39.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:26:39.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:39.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:39.794 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:40.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:40.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:40.796 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:26:40.796 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:26:40.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:40.797 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:41.329 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:42.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:42.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:42.332 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:26:42.332 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:26:42.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:42.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:43.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:26:43.985 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:26:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:43.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:44.444 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:45.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:45.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:45.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:26:45.447 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:26:45.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:45.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:45.874 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:46.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:46.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:46.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:26:46.877 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:26:46.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:46.877 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:47.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:48.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:48.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:48.315 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:26:48.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:26:48.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:48.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:48.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:49.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:49.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:49.908 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:26:49.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:26:49.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:49.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:50.540 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:51.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:51.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:51.543 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:26:51.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:26:51.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:51.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:52.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:53.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:53.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:53.577 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:26:53.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:26:53.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:53.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:54.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:55.685 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:26:55.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:26:55.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:55.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:56.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:57.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:57.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:57.393 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:26:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:26:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:57.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:58.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:59.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:59.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:59.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:26:59.323 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:26:59.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:59.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:59.801 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/137375008,v1:127.0.0.1:6819/137375008] [v2:127.0.0.1:6820/137375008,v1:127.0.0.1:6821/137375008] exists,up 354fb3c9-bd63-46fd-9500-b7bf7626d115 2024-04-25T09:26:59.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:26:59.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:26:59.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:26:59.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:27:00.332 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T09:27:00.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:27:00.352 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:27:00.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:27:00.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:27:00.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:27:01.008 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:27:01.037 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:27:01.067 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:27:01.082 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:27:01.229 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:27:01.258 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:27:01.287 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:27:01.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:27:01.453 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:27:01.481 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:27:01.513 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:27:01.529 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:27:01.699 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:27:01.727 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:27:01.758 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:27:01.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:27:01.926 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:27:01.959 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:27:01.991 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:27:02.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:27:02.009 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:27:02.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:02.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:02.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.481 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:02.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:02.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:02.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:02.883 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.027 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.644 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.870 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.871 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:03.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:03.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:04.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:04.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:04.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:04.280 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:04.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:04.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:04.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:04.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:04.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:27:04.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:27:04.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:27:04.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:27:04.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:27:04.771 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836505 2024-04-25T09:27:04.797 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672974 2024-04-25T09:27:04.823 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509447 2024-04-25T09:27:04.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:27:07.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:07.841 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:27:08.010 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:27:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:27:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:08.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:27:08.174 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:27:08.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:08.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:27:08.332 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:08.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:08.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:27:08.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:27:08.491 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:08.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:27:08.592 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:27:08.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:08.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:08.713 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:08.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:08.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:08.822 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:08.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:08.826 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:08.965 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:08.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:08.969 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.099 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.258 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.262 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.366 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.370 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.475 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.478 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.600 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.710 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.851 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:09.994 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:09.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:09.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.111 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.220 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.337 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.342 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.454 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.571 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.575 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.693 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:10.839 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:10.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:10.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:11.015 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:11.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:27:11.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:27:11.157 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:27:11.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:27:11.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:27:11.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:27:11.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:27:11.161 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:27:11.310 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836509 2024-04-25T09:27:11.334 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672978 2024-04-25T09:27:11.358 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509450 2024-04-25T09:27:11.373 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:27:14.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:14.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:27:14.544 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:27:14.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:27:14.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:14.545 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:27:14.711 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:27:14.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:14.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:27:14.875 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:14.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:14.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:27:14.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:27:15.035 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:15.035 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:27:15.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-04-25T09:27:15.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:27:15.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-04-25T09:27:15.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:27:15.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:27:15.287 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836512 2024-04-25T09:27:15.313 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672980 2024-04-25T09:27:15.338 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509453 2024-04-25T09:27:15.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:27:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:18.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:27:18.526 INFO:tasks.workunit.client.0.smithi031.stdout:22 2024-04-25T09:27:18.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:27:18.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:18.527 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-04-25T09:27:18.701 INFO:tasks.workunit.client.0.smithi031.stdout:22 2024-04-25T09:27:18.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:18.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:27:18.872 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:18.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:27:18.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:27:18.873 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:27:19.039 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:19.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:27:19.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:27:19.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:27:19.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:27:19.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:19.041 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:19.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:19.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:19.041 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:19.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:19.353 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:27:19.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:27:19.355 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:27:19.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:27:19.356 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:27:19.357 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:27:19.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:19.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:27:19.359 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:19.359 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:27:19.360 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:19.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:27:19.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:27:19.382 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:27:19.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:19.383 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:19.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:27:19.385 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:19.386 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:19.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:19.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:19.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:19.389 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:27:19.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:27:19.390 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:27:19.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:27:19.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:27:19.391 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:27:19.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:19.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:27:19.393 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:27:19.393 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:19.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:19.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:27:19.396 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:27:19.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:27:19.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:19.397 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:19.398 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:27:19.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:27:19.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:27:19.399 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:27:19.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:27:19.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:19.400 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:19.401 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.486059 2024-04-25T09:27:19.402 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:27:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:27:19.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:27:19.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:27:19.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:19.855 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:19.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:19.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:19.855 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:19.856 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:27:19.856 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:27:20.065 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:27:20.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:27:20.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:27:20.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:27:20.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:27:20.067 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:27:20.068 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:27:20.068 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:27:20.068 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:27:20.068 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:27:20.069 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:20.069 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:20.069 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:27:20.069 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:27:20.069 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get fsid 2024-04-25T09:27:20.185 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:27:20.186 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:27:20.187 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:27:20.187 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:20.187 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:20.187 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:27:20.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:27:20.187 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get mon_host 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:27:20.298 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:27:21.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:27:21.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:21.214 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:21.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:21.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:21.215 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:21.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:27:21.215 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:27:21.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:27:21.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:27:21.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:27:21.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:27:21.254 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:27:21.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:27:21.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:27:21.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:27:21.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:27:21.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:27:21.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:27:21.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:27:21.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=48ce86a8-d304-4552-a1e7-b7027662723b 2024-04-25T09:27:21.257 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 48ce86a8-d304-4552-a1e7-b7027662723b 2024-04-25T09:27:21.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 48ce86a8-d304-4552-a1e7-b7027662723b' 2024-04-25T09:27:21.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:27:21.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5ISpmjdokERAAXFEJ/AIuMG7/D+DnhNMgdA== 2024-04-25T09:27:21.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5ISpmjdokERAAXFEJ/AIuMG7/D+DnhNMgdA=="}' 2024-04-25T09:27:21.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 48ce86a8-d304-4552-a1e7-b7027662723b -i td/repro-long-log/0/new.json 2024-04-25T09:27:21.560 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:21.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:27:21.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5ISpmjdokERAAXFEJ/AIuMG7/D+DnhNMgdA== --osd-uuid 48ce86a8-d304-4552-a1e7-b7027662723b 2024-04-25T09:27:21.601 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.614+0000 7f0ba5adf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:21.628 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.642+0000 7f0ba5adf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:21.630 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.642+0000 7f0ba5adf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:21.630 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.646+0000 7f0ba5adf7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:27:21.631 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.646+0000 7f0ba5adf7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:27:21.631 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.646+0000 7f0ba5adf7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:27:21.669 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:21.682+0000 7f0ba5adf7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:27:27.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:27:27.858 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:27:27.859 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:27:27.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:27:27.860 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:27:28.477 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:27:28.478 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:27:28.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:27:28.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:27:28.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:27:28.479 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:27:28.508 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:28.522+0000 7f614c4767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:28.510 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:28.526+0000 7f614c4767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:28.511 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:28.526+0000 7f614c4767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:28.971 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:27:28.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:27:28.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:28.972 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:29.056 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:29.070+0000 7f614c4767c0 -1 Falling back to public interface 2024-04-25T09:27:29.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:30.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:30.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:30.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:27:30.448 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:27:30.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:30.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:30.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:31.582 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:31.594+0000 7f614c4767c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:27:31.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:31.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:31.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:27:31.915 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:27:31.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:31.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:32.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:33.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:33.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:33.541 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:27:33.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:27:33.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:33.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:34.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:35.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:35.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:35.009 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:27:35.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:27:35.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:35.010 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:35.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:36.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:36.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:36.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:27:36.471 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:27:36.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:36.472 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:36.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:37.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:37.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:27:37.930 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:27:37.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:37.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:38.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:39.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:39.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:39.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:27:39.452 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:27:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:39.452 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:39.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:40.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:40.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:40.892 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:27:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:27:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:40.892 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:41.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:42.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:42.451 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:27:42.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:42.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:27:42.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:42.451 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:42.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:43.895 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:43.895 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:27:43.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:43.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:27:43.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:43.896 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:44.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:45.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:45.323 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:27:45.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:45.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:27:45.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:45.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:46.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:47.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:47.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:47.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:27:47.286 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:27:47.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:47.287 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:48.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:49.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:49.142 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:49.143 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:27:49.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:27:49.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:49.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:49.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:50.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:50.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:50.960 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:27:50.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:27:50.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:50.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:51.982 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:52.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:52.984 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:27:52.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:52.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:27:52.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:52.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:53.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:54.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:54.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:54.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:27:54.682 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:27:54.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:54.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:55.112 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3458726617,v1:127.0.0.1:6803/3458726617] [v2:127.0.0.1:6804/3458726617,v1:127.0.0.1:6805/3458726617] exists,up 48ce86a8-d304-4552-a1e7-b7027662723b 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:27:55.116 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:55.117 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:27:55.118 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:27:55.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:27:55.119 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:27:55.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7caad067-8842-4b22-87d1-36789665ce94 2024-04-25T09:27:55.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7caad067-8842-4b22-87d1-36789665ce94' 2024-04-25T09:27:55.120 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 7caad067-8842-4b22-87d1-36789665ce94 2024-04-25T09:27:55.120 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:27:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbIipm13lOCRAAhxGcb8R9k2eVjkr4EcAcjw== 2024-04-25T09:27:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbIipm13lOCRAAhxGcb8R9k2eVjkr4EcAcjw=="}' 2024-04-25T09:27:55.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7caad067-8842-4b22-87d1-36789665ce94 -i td/repro-long-log/1/new.json 2024-04-25T09:27:55.630 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:27:55.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:27:55.647 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbIipm13lOCRAAhxGcb8R9k2eVjkr4EcAcjw== --osd-uuid 7caad067-8842-4b22-87d1-36789665ce94 2024-04-25T09:27:55.680 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.694+0000 7f484e0447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:55.683 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.698+0000 7f484e0447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.698+0000 7f484e0447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.698+0000 7f484e0447c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:27:55.684 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.698+0000 7f484e0447c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:27:55.685 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.698+0000 7f484e0447c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:27:55.726 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:27:55.738+0000 7f484e0447c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:28:01.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:28:01.711 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:28:01.712 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:28:01.712 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:28:01.713 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:28:02.293 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:28:02.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:28:02.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:28:02.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:28:02.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:28:02.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:28:02.324 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:02.337+0000 7f3b6c6357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:02.327 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:02.341+0000 7f3b6c6357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:02.328 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:02.341+0000 7f3b6c6357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:02.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:28:02.762 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:28:02.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:02.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:02.876 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:02.889+0000 7f3b6c6357c0 -1 Falling back to public interface 2024-04-25T09:28:03.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:04.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:04.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:04.241 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:28:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:28:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:04.242 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:04.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:05.406 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:05.417+0000 7f3b6c6357c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:28:05.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:05.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:05.722 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:28:05.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:28:05.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:05.723 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:06.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:07.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:07.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:07.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:28:07.215 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:28:07.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:07.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:07.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:08.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:08.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:08.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:28:08.654 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:28:08.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:08.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:09.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:10.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:10.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:10.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:28:10.205 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:28:10.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:10.206 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:10.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:11.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:11.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:11.721 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:28:11.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:28:11.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:11.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:12.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:13.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:13.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:13.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:28:13.215 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:28:13.215 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:13.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:13.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:14.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:14.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:14.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:28:14.655 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:28:14.655 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:14.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:15.264 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:16.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:16.266 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:28:16.267 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:28:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:16.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:16.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:17.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:17.754 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:17.754 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:28:17.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:28:17.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:17.755 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:18.192 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:19.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:19.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:19.195 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:28:19.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:28:19.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:19.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:19.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:20.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:20.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:20.907 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:28:20.908 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:28:20.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:20.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:21.547 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:22.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:22.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:22.549 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:28:22.549 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:28:22.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:22.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:23.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:28:24.213 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:28:24.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:24.213 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:24.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:25.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:25.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:25.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:28:25.801 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:28:25.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:25.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:26.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:27.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:27.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:27.312 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:28:27.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:28:27.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:27.313 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:27.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:28.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:28.762 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:28.763 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T09:28:28.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:28:28.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:28.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:29.232 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1823555704,v1:127.0.0.1:6811/1823555704] [v2:127.0.0.1:6812/1823555704,v1:127.0.0.1:6813/1823555704] exists,up 7caad067-8842-4b22-87d1-36789665ce94 2024-04-25T09:28:29.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:28:29.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:28:29.237 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:28:29.238 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:28:29.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:28:29.240 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:28:29.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8fec322-4727-4968-8270-3abb499ec14e 2024-04-25T09:28:29.240 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 a8fec322-4727-4968-8270-3abb499ec14e 2024-04-25T09:28:29.240 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a8fec322-4727-4968-8270-3abb499ec14e' 2024-04-25T09:28:29.241 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:28:29.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9IipmfCwqEBAAe1Nn10+emaRnNpZmTcRbbA== 2024-04-25T09:28:29.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9IipmfCwqEBAAe1Nn10+emaRnNpZmTcRbbA=="}' 2024-04-25T09:28:29.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8fec322-4727-4968-8270-3abb499ec14e -i td/repro-long-log/2/new.json 2024-04-25T09:28:29.747 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:28:29.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:28:29.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9IipmfCwqEBAAe1Nn10+emaRnNpZmTcRbbA== --osd-uuid a8fec322-4727-4968-8270-3abb499ec14e 2024-04-25T09:28:29.799 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.813+0000 7f41d85937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:29.801 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.813+0000 7f41d85937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.817+0000 7f41d85937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.817+0000 7f41d85937c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:28:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.817+0000 7f41d85937c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:28:29.803 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.817+0000 7f41d85937c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:28:29.888 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:29.901+0000 7f41d85937c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:28:35.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:28:35.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:28:35.945 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:28:35.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:28:35.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:28:36.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:28:36.550 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:28:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:28:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:28:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:28:36.551 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:28:36.580 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:36.593+0000 7f02334e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:36.582 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:36.597+0000 7f02334e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:36.583 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:36.597+0000 7f02334e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:36.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:28:36.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:28:36.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:28:36.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:28:36.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:28:36.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:36.992 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:28:36.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:28:36.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:36.993 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:37.139 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:37.153+0000 7f02334e77c0 -1 Falling back to public interface 2024-04-25T09:28:37.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:38.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:38.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:38.465 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:28:38.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:28:38.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:38.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:38.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:39.644 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:28:39.657+0000 7f02334e77c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:28:39.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:39.887 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:39.887 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:28:39.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:28:39.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:39.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:40.520 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:41.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:41.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:41.523 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:28:41.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:28:41.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:41.523 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:41.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:42.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:42.986 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:42.987 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:28:42.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:28:42.987 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:42.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:43.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:44.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:44.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:44.471 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:28:44.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:28:44.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:44.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:45.003 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:46.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:46.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:46.006 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:28:46.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:28:46.006 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:46.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:46.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:47.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:47.555 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:28:47.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:47.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:28:47.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:47.556 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:48.005 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:49.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:49.008 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:28:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:28:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:49.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:49.555 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:50.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:50.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:50.557 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:28:50.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:28:50.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:50.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:50.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:51.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:51.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:51.991 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:28:51.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:28:51.991 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:51.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:52.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:53.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:53.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:53.665 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:28:53.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:28:53.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:53.666 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:55.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:56.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:56.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:56.169 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:28:56.170 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:28:56.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:56.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:56.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:57.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:57.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:57.597 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:28:57.598 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:28:57.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:57.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:28:58.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:59.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:59.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:59.296 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:28:59.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:28:59.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:59.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:00.014 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:01.015 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:01.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:01.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:29:01.016 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:29:01.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:01.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:01.985 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:02.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:02.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:02.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:29:02.988 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:29:02.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:02.989 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:03.427 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3641898959,v1:127.0.0.1:6819/3641898959] [v2:127.0.0.1:6820/3641898959,v1:127.0.0.1:6821/3641898959] exists,up a8fec322-4727-4968-8270-3abb499ec14e 2024-04-25T09:29:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:29:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:29:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:29:03.428 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:29:03.974 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T09:29:03.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:29:03.993 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:29:04.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:29:04.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:29:04.456 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:29:04.618 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:29:04.649 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:29:04.679 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:29:04.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:29:04.861 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:29:04.890 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:29:04.922 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:29:04.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:29:05.084 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:29:05.112 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:29:05.141 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:29:05.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:29:05.342 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:29:05.371 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:29:05.401 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:29:05.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:29:05.569 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:29:05.598 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:29:05.627 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:29:05.643 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:29:05.645 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:29:05.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:05.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:05.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:05.997 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.114 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.566 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.842 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:06.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:06.960 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.186 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.312 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.454 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.688 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:07.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:07.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:08.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:08.073 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:08.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:08.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:08.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:08.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:29:08.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:29:08.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:29:08.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:29:08.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:29:08.500 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:29:08.650 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836503 2024-04-25T09:29:08.675 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672971 2024-04-25T09:29:08.699 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509449 2024-04-25T09:29:08.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:29:11.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:11.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:29:11.889 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:29:11.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:11.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:29:11.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:29:12.044 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:29:12.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:12.045 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:29:12.224 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:12.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:12.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:29:12.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:29:12.387 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:12.387 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:29:12.485 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:29:12.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:12.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:12.599 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:12.750 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:12.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:12.753 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:12.883 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:12.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:12.890 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:12.992 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:12.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:12.996 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.101 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.235 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.343 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.494 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.497 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.603 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.754 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.871 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.875 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:13.988 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:13.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:13.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.105 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.110 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.264 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.381 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.384 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.557 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.675 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.800 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:14.926 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:14.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:29:14.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:29:15.068 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:29:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:29:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:29:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:29:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:29:15.072 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:29:15.223 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836507 2024-04-25T09:29:15.247 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672975 2024-04-25T09:29:15.273 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509452 2024-04-25T09:29:15.288 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:29:18.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:18.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:29:18.459 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:29:18.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:18.459 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:29:18.460 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:29:18.624 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:29:18.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:18.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:29:18.782 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:18.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:18.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:29:18.783 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:29:18.936 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:18.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-04-25T09:29:18.937 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-04-25T09:29:19.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-04-25T09:29:19.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-04-25T09:29:19.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:29:19.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:29:19.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:29:19.104 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:29:19.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:29:19.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:29:19.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-04-25T09:29:19.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-04-25T09:29:20.816 INFO:tasks.workunit.client.0.smithi031.stderr:Log bounds are: (19'20,19'41] 2024-04-25T09:29:20.816 INFO:tasks.workunit.client.0.smithi031.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-04-25T09:29:20.844 INFO:tasks.workunit.client.0.smithi031.stderr:Finished trimming, now compacting... 2024-04-25T09:29:21.682 INFO:tasks.workunit.client.0.smithi031.stdout:Finished trimming pg log 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:29:22.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:29:22.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:29:22.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:29:22.103 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:29:22.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:29:22.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:22.103 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:22.104 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:22.104 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:22.104 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:22.104 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:29:22.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:29:22.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T09:29:22.106 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:29:22.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:29:22.106 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-04-25T09:29:22.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T09:29:22.108 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:29:22.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:29:22.109 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:29:22.140 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:22.152+0000 7f84518b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:22.143 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:22.156+0000 7f84518b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:22.145 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:22.156+0000 7f84518b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:29:22.564 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:22.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:22.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:22.695 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:22.708+0000 7f84518b37c0 -1 Falling back to public interface 2024-04-25T09:29:23.058 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:24.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:24.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:24.060 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:29:24.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:29:24.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:24.061 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:24.558 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:25.048 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:25.060+0000 7f84518b37c0 -1 osd.1 19 log_to_monitors true 2024-04-25T09:29:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:29:25.560 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:29:25.560 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:25.561 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:25.998 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/3662172146,v1:127.0.0.1:6811/3662172146] [v2:127.0.0.1:6812/3662172146,v1:127.0.0.1:6813/3662172146] exists,up 7caad067-8842-4b22-87d1-36789665ce94 2024-04-25T09:29:25.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:29:25.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:29:25.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:29:25.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-04-25T09:29:25.999 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:29:26.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:29:26.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:29:26.000 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:29:26.001 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:29:26.001 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:29:26.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:29:26.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:29:26.001 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:29:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:29:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:29:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:29:26.111 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:29:26.112 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:29:26.112 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:29:26.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:29:26.584 INFO:tasks.workunit.client.0.smithi031.stderr:1 2024-04-25T09:29:26.585 INFO:tasks.workunit.client.0.smithi031.stderr:2' 2024-04-25T09:29:26.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:29:26.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:26.585 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:29:26.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T09:29:26.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T09:29:26.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T09:29:26.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:26.720 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:29:26.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=98784247811 2024-04-25T09:29:26.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 98784247811 2024-04-25T09:29:26.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-98784247811' 2024-04-25T09:29:26.872 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:26.872 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:29:27.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-04-25T09:29:27.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-04-25T09:29:27.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-98784247811 2-64424509456' 2024-04-25T09:29:27.028 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:27.028 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T09:29:27.029 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:27.030 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:29:27.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T09:29:27.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:27.031 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T09:29:27.031 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.0 seq 21474836511 2024-04-25T09:29:27.032 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T09:29:27.032 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:27.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T09:29:27.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:29:28.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:29:28.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:28.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T09:29:28.876 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:28.876 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-98784247811 2024-04-25T09:29:28.877 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:28.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:29:28.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-98784247811 2024-04-25T09:29:28.879 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:28.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=98784247811 2024-04-25T09:29:28.881 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.1 seq 98784247811 2024-04-25T09:29:28.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-04-25T09:29:28.882 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:29:29.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247811 -lt 98784247811 2024-04-25T09:29:29.315 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:29.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-04-25T09:29:29.315 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:29.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:29:29.317 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-04-25T09:29:29.318 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:29.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-04-25T09:29:29.319 INFO:tasks.workunit.client.0.smithi031.stdout:waiting osd.2 seq 64424509456 2024-04-25T09:29:29.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-25T09:29:29.319 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:29:29.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509456 2024-04-25T09:29:29.782 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:29:29.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:29:29.783 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:29:30.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:29:30.362 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:29:30.767 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:29:30.767 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:29:30.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:29:30.768 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:29:31.376 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:29:31.513 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836514 2024-04-25T09:29:31.537 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 98784247814 2024-04-25T09:29:31.561 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509459 2024-04-25T09:29:31.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:29:34.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:34.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:29:34.746 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:29:34.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:34.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:29:34.747 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:29:34.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:29:34.906 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:35.061 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:29:35.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:29:35.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:29:35.063 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:29:35.218 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:29:35.219 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:29:35.220 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:29:35.530 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:29:35.530 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:29:35.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:29:35.532 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:29:35.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:29:35.533 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:29:35.534 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:29:35.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:29:35.535 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:29:35.536 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:29:35.536 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:29:35.538 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:29:35.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:29:35.539 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:29:35.562 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:29:35.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:35.563 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:35.563 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:29:35.564 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:29:35.565 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:29:35.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:29:35.568 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:29:35.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:29:35.569 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:29:35.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:29:35.570 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:29:35.570 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:29:35.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:29:35.571 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:29:35.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:29:35.572 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:29:35.573 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:29:35.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:29:35.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:29:35.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:29:35.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:35.576 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:35.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:29:35.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:29:35.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:29:35.577 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:29:35.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:29:35.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:35.579 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:35.579 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.486059 2024-04-25T09:29:35.581 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:29:35.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:29:35.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:29:36.051 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:29:36.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:36.052 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:36.052 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:36.052 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:36.052 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:36.053 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:29:36.053 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:29:36.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:29:36.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:29:36.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:29:36.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:29:36.256 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:29:36.257 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:29:36.257 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:29:36.257 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:29:36.257 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:29:36.257 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:29:36.258 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:36.258 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:36.258 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:29:36.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:29:36.258 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get fsid 2024-04-25T09:29:36.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:29:36.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:29:36.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:29:36.377 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:29:36.378 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:29:36.378 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:29:36.378 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:29:36.378 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:29:36.378 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:29:36.379 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:36.379 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:36.379 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:29:36.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:29:36.379 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get mon_host 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:29:36.499 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:29:37.421 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:29:37.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:37.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:37.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:37.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:37.422 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:37.422 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:29:37.423 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:29:37.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:29:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:29:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:29:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:29:37.491 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:29:37.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16028f65-9b15-42c0-aaac-1c446edd661d 2024-04-25T09:29:37.491 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 16028f65-9b15-42c0-aaac-1c446edd661d 2024-04-25T09:29:37.491 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 16028f65-9b15-42c0-aaac-1c446edd661d' 2024-04-25T09:29:37.492 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:29:37.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBIipmmNE9HxAAt/sIUuuIg8AZag9vAPTgSQ== 2024-04-25T09:29:37.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBIipmmNE9HxAAt/sIUuuIg8AZag9vAPTgSQ=="}' 2024-04-25T09:29:37.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16028f65-9b15-42c0-aaac-1c446edd661d -i td/repro-long-log/0/new.json 2024-04-25T09:29:37.800 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:37.816 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:29:37.818 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBIipmmNE9HxAAt/sIUuuIg8AZag9vAPTgSQ== --osd-uuid 16028f65-9b15-42c0-aaac-1c446edd661d 2024-04-25T09:29:37.854 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.868+0000 7f473b23b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:37.869 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.880+0000 7f473b23b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:37.870 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.884+0000 7f473b23b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.884+0000 7f473b23b7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:29:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.884+0000 7f473b23b7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:29:37.871 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.884+0000 7f473b23b7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:29:37.908 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:37.920+0000 7f473b23b7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:29:44.179 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:29:44.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:29:44.180 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:29:44.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:29:44.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:29:44.755 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:29:44.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:29:44.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:29:44.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:29:44.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:29:44.756 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:29:44.785 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:44.800+0000 7f04f36697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:44.791 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:44.808+0000 7f04f36697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:44.792 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:44.808+0000 7f04f36697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:45.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:29:45.223 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:29:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:45.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:45.384 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:45.400+0000 7f04f36697c0 -1 Falling back to public interface 2024-04-25T09:29:45.668 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:46.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:46.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:46.669 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:29:46.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:29:46.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:46.670 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:47.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:47.963 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:29:47.980+0000 7f04f36697c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:29:48.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:48.141 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:29:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:29:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:48.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:48.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:49.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:49.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:49.708 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:29:49.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:29:49.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:49.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:50.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:51.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:51.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:51.151 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:29:51.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:29:51.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:51.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:51.725 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:52.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:52.728 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:29:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:29:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:52.728 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:53.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:54.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:54.171 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:29:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:29:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:54.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:54.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:55.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:55.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:55.742 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:29:55.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:29:55.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:55.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:56.199 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:57.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:57.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:57.201 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:29:57.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:29:57.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:57.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:57.704 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:58.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:58.706 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:59.149 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:00.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:00.150 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:00.151 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:30:00.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:30:00.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:00.151 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:00.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:01.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:01.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:01.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:30:01.709 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:30:01.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:01.710 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:02.485 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:03.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:03.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:03.487 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:30:03.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:30:03.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:03.487 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:04.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:05.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:05.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:05.325 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:30:05.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:30:05.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:05.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:06.152 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:07.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:07.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:07.154 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:30:07.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:30:07.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:07.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:07.764 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:08.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:08.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:08.766 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:30:08.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:30:08.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:08.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:09.908 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:10.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:10.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:10.910 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:30:10.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:30:10.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:10.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:11.461 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/743387538,v1:127.0.0.1:6803/743387538] [v2:127.0.0.1:6804/743387538,v1:127.0.0.1:6805/743387538] exists,up 16028f65-9b15-42c0-aaac-1c446edd661d 2024-04-25T09:30:11.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:30:11.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:30:11.466 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:30:11.467 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:30:11.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c6bc4d5-3790-4c54-8f6a-80b7015e7806 2024-04-25T09:30:11.468 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 5c6bc4d5-3790-4c54-8f6a-80b7015e7806 2024-04-25T09:30:11.468 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5c6bc4d5-3790-4c54-8f6a-80b7015e7806' 2024-04-25T09:30:11.469 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:30:11.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjIipmIUS/HRAAyAXLCghV+OLl3OQTKs1nxw== 2024-04-25T09:30:11.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjIipmIUS/HRAAyAXLCghV+OLl3OQTKs1nxw=="}' 2024-04-25T09:30:11.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c6bc4d5-3790-4c54-8f6a-80b7015e7806 -i td/repro-long-log/1/new.json 2024-04-25T09:30:12.000 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:30:12.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:30:12.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjIipmIUS/HRAAyAXLCghV+OLl3OQTKs1nxw== --osd-uuid 5c6bc4d5-3790-4c54-8f6a-80b7015e7806 2024-04-25T09:30:12.050 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.063+0000 7fa570c817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:12.052 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.067+0000 7fa570c817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:12.053 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.067+0000 7fa570c817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:12.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.067+0000 7fa570c817c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:30:12.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.067+0000 7fa570c817c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:30:12.054 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.067+0000 7fa570c817c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:30:12.093 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:12.107+0000 7fa570c817c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:30:18.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:30:18.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:30:18.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:30:18.175 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:30:18.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:30:18.810 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:30:18.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:30:18.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:30:18.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:30:18.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:30:18.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:30:18.840 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:18.855+0000 7f4542dc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:18.843 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:18.859+0000 7f4542dc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:18.844 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:18.859+0000 7f4542dc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:19.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:30:19.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:30:19.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:30:19.295 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:30:19.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:30:19.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:19.296 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:30:19.296 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:30:19.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:19.297 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:19.388 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:19.403+0000 7f4542dc67c0 -1 Falling back to public interface 2024-04-25T09:30:19.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:20.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:20.774 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:20.775 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:30:20.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:30:20.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:20.775 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:21.200 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:21.895 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:21.911+0000 7f4542dc67c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:30:22.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:22.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:22.202 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:30:22.202 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:30:22.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:22.203 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:22.749 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:23.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:23.751 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:30:23.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:23.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:30:23.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:23.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:24.180 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:25.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:25.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:25.183 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:30:25.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:30:25.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:25.183 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:25.682 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:26.683 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:26.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:26.684 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:30:26.684 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:30:26.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:26.685 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:27.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:28.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:28.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:28.122 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:30:28.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:30:28.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:28.122 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:28.727 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:29.729 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:29.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:29.730 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:30:29.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:30:29.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:29.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:30.164 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:31.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:31.166 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:31.166 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:30:31.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:30:31.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:31.167 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:31.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:32.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:32.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:32.702 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:30:32.702 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:30:32.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:32.703 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:33.253 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:34.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:34.255 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:34.256 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:30:34.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:30:34.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:34.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:35.121 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:36.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:36.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:36.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:30:36.124 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:30:36.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:36.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:36.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:37.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:37.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:37.926 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:30:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:30:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:37.927 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:38.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:39.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:39.679 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:30:39.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:30:39.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:39.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:40.462 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:41.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:41.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:41.464 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:30:41.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:30:41.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:41.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:42.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:43.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:43.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:43.410 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:30:43.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:30:43.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:43.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:44.282 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:45.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:45.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:45.284 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:30:45.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:30:45.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:45.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:30:45.753 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/611900255,v1:127.0.0.1:6811/611900255] [v2:127.0.0.1:6812/611900255,v1:127.0.0.1:6813/611900255] exists,up 5c6bc4d5-3790-4c54-8f6a-80b7015e7806 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:30:45.757 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:30:45.758 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:30:45.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:30:45.760 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc0ec833-4b06-45f0-97b0-1c54c7c09ca4 2024-04-25T09:30:45.761 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dc0ec833-4b06-45f0-97b0-1c54c7c09ca4' 2024-04-25T09:30:45.761 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 dc0ec833-4b06-45f0-97b0-1c54c7c09ca4 2024-04-25T09:30:45.761 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:30:45.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFIipmoxaDLxAAO9z3iZWP4tYkOLt4Tpq2Hw== 2024-04-25T09:30:45.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFIipmoxaDLxAAO9z3iZWP4tYkOLt4Tpq2Hw=="}' 2024-04-25T09:30:45.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc0ec833-4b06-45f0-97b0-1c54c7c09ca4 -i td/repro-long-log/2/new.json 2024-04-25T09:30:46.328 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:30:46.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:30:46.347 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFIipmoxaDLxAAO9z3iZWP4tYkOLt4Tpq2Hw== --osd-uuid dc0ec833-4b06-45f0-97b0-1c54c7c09ca4 2024-04-25T09:30:46.373 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.387+0000 7fd3c77587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:46.402 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.419+0000 7fd3c77587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:46.403 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.419+0000 7fd3c77587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:46.404 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.419+0000 7fd3c77587c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:30:46.404 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.419+0000 7fd3c77587c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:30:46.404 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.419+0000 7fd3c77587c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:30:46.510 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:46.523+0000 7fd3c77587c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:30:52.810 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:30:52.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:30:52.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:30:52.812 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:30:52.813 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:30:53.399 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:30:53.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:30:53.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:30:53.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:30:53.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:30:53.400 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:30:53.431 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:53.447+0000 7f2b8ca1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:53.449 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:53.463+0000 7f2b8ca1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:53.450 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:53.467+0000 7f2b8ca1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:53.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:30:53.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:30:53.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:30:53.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:30:53.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:30:53.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:53.882 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:30:53.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:30:53.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:53.882 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:54.011 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:54.027+0000 7f2b8ca1b7c0 -1 Falling back to public interface 2024-04-25T09:30:54.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:55.317 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:55.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:55.318 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:30:55.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:30:55.318 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:55.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:55.750 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:56.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:56.751 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:56.752 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:30:56.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:30:56.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:56.752 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:56.780 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:30:56.795+0000 7f2b8ca1b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:30:57.346 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:58.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:58.348 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:58.349 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:30:58.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:30:58.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:58.349 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:58.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:59.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:59.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:59.947 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:30:59.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:30:59.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:59.948 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:00.389 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:01.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:01.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:01.391 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:31:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:31:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:01.392 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:01.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:02.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:02.832 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:02.832 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:31:02.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:31:02.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:02.833 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:03.405 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:04.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:04.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:04.407 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:31:04.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:31:04.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:04.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:04.836 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:05.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:05.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:05.839 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:31:05.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:31:05.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:05.839 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:06.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:07.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:07.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:07.291 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:31:07.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:31:07.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:07.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:07.785 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:08.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:08.787 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:08.787 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:31:08.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:31:08.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:08.788 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:09.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:10.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:10.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:10.339 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:31:10.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:31:10.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:10.340 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:10.878 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:11.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:11.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:11.880 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:31:11.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:31:11.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:11.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:12.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:13.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:13.568 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:13.569 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:31:13.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:31:13.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:13.569 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:14.256 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:15.257 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:15.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:15.258 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:31:15.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:31:15.258 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:15.259 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:15.974 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:16.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:16.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:16.977 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:31:16.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:31:16.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:16.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:18.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:19.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:19.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:19.144 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:31:19.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:31:19.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:19.145 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:19.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:20.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:20.721 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:20.721 INFO:tasks.workunit.client.0.smithi031.stdout:17 2024-04-25T09:31:20.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:31:20.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:20.722 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:31:21.187 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1097130214,v1:127.0.0.1:6819/1097130214] [v2:127.0.0.1:6820/1097130214,v1:127.0.0.1:6821/1097130214] exists,up dc0ec833-4b06-45f0-97b0-1c54c7c09ca4 2024-04-25T09:31:21.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:31:21.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:31:21.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:31:21.188 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:31:21.696 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T09:31:21.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:31:21.715 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:31:22.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:31:22.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:31:22.201 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:31:22.352 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:22.380 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:22.408 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:22.422 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:31:22.561 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:22.589 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:22.617 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:22.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:31:22.773 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:22.802 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:22.831 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:22.845 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:31:22.984 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:23.012 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:23.039 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:23.054 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:31:23.235 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:23.266 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:23.302 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:23.319 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:31:23.321 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:31:23.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:23.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:23.582 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:23.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:23.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:23.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:23.911 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.194 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.195 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.303 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.562 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.805 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:24.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:24.926 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.040 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.651 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:25.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:25.885 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:31:26.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:31:26.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:26.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:31:26.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:26.029 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:26.179 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836502 2024-04-25T09:31:26.203 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672978 2024-04-25T09:31:26.229 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509449 2024-04-25T09:31:26.244 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:29.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:29.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:29.415 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:31:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:31:29.571 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:31:29.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:29.572 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:29.730 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:29.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:29.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:29.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:29.880 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:29.881 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.038 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:31:30.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.039 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.142 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.146 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.243 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.247 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.377 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.381 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.494 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.611 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.754 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.759 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:30.905 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:30.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:30.909 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.005 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.009 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.122 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.125 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.223 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.226 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.323 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.432 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.583 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.691 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.695 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.696 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.808 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.812 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:31.909 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:31.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:31.913 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:32.018 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:32.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:32.022 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:32.127 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:32.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:32.130 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:32.235 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:32.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:31:32.239 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:31:32.386 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:32.390 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:31:32.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:32.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:31:32.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:32.391 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:32.565 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836506 2024-04-25T09:31:32.590 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672981 2024-04-25T09:31:32.615 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509452 2024-04-25T09:31:32.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:35.632 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:35.801 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:31:35.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:35.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:35.802 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:31:35.960 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:31:35.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:35.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:36.112 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:36.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:36.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:36.113 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:36.265 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:36.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:31:36.409 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:36.438 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:36.467 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:36.482 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:31:36.638 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:36.666 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:36.694 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:36.709 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:31:36.857 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:36.885 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:36.915 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:36.930 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:31:37.066 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:37.097 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:37.125 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:37.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-04-25T09:31:37.280 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:31:37.308 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:31:37.339 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:31:37.353 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:31:37.467 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-04-25T09:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:37.471 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:37.613 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836509 2024-04-25T09:31:37.637 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672984 2024-04-25T09:31:37.662 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509455 2024-04-25T09:31:37.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:40.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:40.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:40.853 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:31:40.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:40.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:40.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:31:41.007 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:31:41.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:41.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:41.173 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:41.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:41.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:41.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:41.333 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:41.333 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:31:41.436 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:41.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-04-25T09:31:41.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:41.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:31:41.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:41.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:41.587 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836512 2024-04-25T09:31:41.612 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672987 2024-04-25T09:31:41.637 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509458 2024-04-25T09:31:41.652 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:44.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:44.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:44.826 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:31:44.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:44.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:44.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:31:44.983 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:31:44.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:44.984 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:45.131 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:45.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:45.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:45.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:45.283 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:45.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:31:45.422 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:45.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-04-25T09:31:45.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:45.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:31:45.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:45.426 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:45.573 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836514 2024-04-25T09:31:45.596 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672990 2024-04-25T09:31:45.621 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509461 2024-04-25T09:31:45.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:48.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:48.638 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:48.805 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:31:48.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:48.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:48.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:31:48.960 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:31:48.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:48.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:49.117 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:49.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:49.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:49.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:49.277 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:49.277 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:31:49.382 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:49.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-04-25T09:31:49.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:49.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:31:49.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:49.386 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:49.534 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836517 2024-04-25T09:31:49.558 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672992 2024-04-25T09:31:49.583 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509463 2024-04-25T09:31:49.598 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:52.601 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:52.602 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:52.772 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:31:52.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:52.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:52.773 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:31:52.937 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:31:52.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:52.938 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:53.094 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:53.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:53.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:53.095 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:53.254 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:53.254 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:31:53.410 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:53.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-04-25T09:31:53.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:53.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:31:53.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:53.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:53.559 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836520 2024-04-25T09:31:53.584 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672995 2024-04-25T09:31:53.608 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509466 2024-04-25T09:31:53.623 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:31:56.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:56.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:56.794 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:31:56.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:56.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:31:56.795 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:31:56.958 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:31:56.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:56.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:57.116 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:57.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:31:57.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:31:57.117 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:31:57.279 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:31:57.279 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:31:57.429 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:31:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:31:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:31:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:31:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:31:57.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:31:57.581 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836523 2024-04-25T09:31:57.606 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672998 2024-04-25T09:31:57.630 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509469 2024-04-25T09:31:57.646 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:32:00.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:00.648 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:00.819 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:00.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:00.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:32:00.820 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:00.987 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:00.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:00.988 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:01.146 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:01.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:01.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:01.147 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:32:01.309 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:01.309 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:32:01.414 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:32:01.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:32:01.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:32:01.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:32:01.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:32:01.419 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:32:01.565 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836526 2024-04-25T09:32:01.590 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673001 2024-04-25T09:32:01.615 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509472 2024-04-25T09:32:01.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:32:04.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:04.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:04.802 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:32:04.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:04.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:32:04.803 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:04.966 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:32:04.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:04.966 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:05.123 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:05.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:05.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:32:05.124 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:05.290 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:05.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:32:05.391 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:32:05.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:32:05.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:32:05.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:32:05.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:32:05.395 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:32:05.538 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836528 2024-04-25T09:32:05.562 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673004 2024-04-25T09:32:05.590 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509475 2024-04-25T09:32:05.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:32:08.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:08.607 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:08.778 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:08.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:08.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:08.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:32:08.941 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:08.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:08.942 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:09.105 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:09.105 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:09.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:32:09.271 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:09.271 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:32:09.410 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:32:09.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:32:09.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:32:09.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:32:09.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:32:09.414 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:32:09.563 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836531 2024-04-25T09:32:09.587 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673006 2024-04-25T09:32:09.611 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509477 2024-04-25T09:32:09.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:32:12.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:12.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:12.798 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:32:12.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:12.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:12.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:32:12.962 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:32:12.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:12.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:13.119 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:13.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:13.119 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:13.120 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:32:13.272 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:13.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:32:13.404 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:32:13.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:32:13.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:32:13.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:32:13.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:32:13.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:32:13.546 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836534 2024-04-25T09:32:13.571 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673009 2024-04-25T09:32:13.595 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509480 2024-04-25T09:32:13.611 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:32:16.613 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:16.614 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:16.783 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:16.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:16.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:32:16.784 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:32:16.942 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:16.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:17.106 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:17.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:32:17.106 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:32:17.107 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:32:17.271 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:17.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:32:17.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:32:17.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:32:17.272 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:32:17.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:32:17.272 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:32:17.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:32:17.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:32:17.273 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:32:17.583 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:32:17.583 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:32:17.585 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:32:17.585 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:32:17.586 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:32:17.587 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:32:17.587 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:32:17.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:32:17.588 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:32:17.589 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:32:17.589 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:32:17.590 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:32:17.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:32:17.591 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:32:17.615 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:32:17.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:17.616 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:17.616 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:32:17.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:32:17.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:32:17.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:32:17.620 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:32:17.621 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:32:17.621 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:32:17.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:32:17.622 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:32:17.623 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:32:17.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:32:17.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:32:17.624 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:32:17.625 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:32:17.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:32:17.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:32:17.627 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:32:17.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:32:17.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:17.628 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:17.628 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:32:17.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:32:17.629 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:32:17.630 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:32:17.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:32:17.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:17.631 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:17.631 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.486059 2024-04-25T09:32:17.634 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:32:17.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:32:17.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:32:17.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:32:17.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-04-25T09:32:17.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:32:17.635 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:32:18.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:32:18.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:32:18.075 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:32:18.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:32:18.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:18.076 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:18.076 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:32:18.077 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:32:18.286 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:32:18.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:32:18.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:32:18.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:32:18.288 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:32:18.289 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:32:18.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get fsid 2024-04-25T09:32:18.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:32:18.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:32:18.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:32:18.405 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:32:18.406 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:32:18.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:32:18.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:32:18.406 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:32:18.406 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:32:18.406 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:18.407 INFO:tasks.workunit.client.0.smithi031.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:18.407 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.486059/ceph-mon.a.asok 2024-04-25T09:32:18.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:32:18.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.486059/ceph-mon.a.asok config get mon_host 2024-04-25T09:32:18.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:32:18.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:32:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:32:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:32:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:32:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:32:18.517 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:19.407 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:32:19.408 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:32:19.409 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:32:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:32:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:32:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:32:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:32:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:32:19.446 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:32:19.447 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:32:19.448 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:32:19.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:32:19.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:32:19.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:32:19.449 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:32:19.449 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36036f22-a3a4-4175-b55c-7114aebc4780 2024-04-25T09:32:19.450 INFO:tasks.workunit.client.0.smithi031.stdout:add osd0 36036f22-a3a4-4175-b55c-7114aebc4780 2024-04-25T09:32:19.450 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 36036f22-a3a4-4175-b55c-7114aebc4780' 2024-04-25T09:32:19.450 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:32:19.464 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjIypmQbG5HBAAgD3fLMLq6hRRx6Ow/7M9+g== 2024-04-25T09:32:19.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjIypmQbG5HBAAgD3fLMLq6hRRx6Ow/7M9+g=="}' 2024-04-25T09:32:19.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36036f22-a3a4-4175-b55c-7114aebc4780 -i td/repro-long-log/0/new.json 2024-04-25T09:32:19.757 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:19.771 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:32:19.772 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjIypmQbG5HBAAgD3fLMLq6hRRx6Ow/7M9+g== --osd-uuid 36036f22-a3a4-4175-b55c-7114aebc4780 2024-04-25T09:32:19.808 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.817+0000 7f6845dca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:19.825 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.837+0000 7f6845dca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:19.826 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.841+0000 7f6845dca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:19.826 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.841+0000 7f6845dca7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:32:19.826 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.841+0000 7f6845dca7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:32:19.827 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.841+0000 7f6845dca7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:32:19.864 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:19.877+0000 7f6845dca7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:32:26.016 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:32:26.017 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:32:26.017 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd0 key to auth repository 2024-04-25T09:32:26.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:32:26.018 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:32:26.603 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.0 2024-04-25T09:32:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:32:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:32:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:32:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:32:26.604 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:32:26.635 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:26.649+0000 7f478dfc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:26.645 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:26.657+0000 7f478dfc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:26.646 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:26.661+0000 7f478dfc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:27.059 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:32:27.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:32:27.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:27.060 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:27.227 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:27.241+0000 7f478dfc07c0 -1 Falling back to public interface 2024-04-25T09:32:27.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:28.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:28.542 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:28.543 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:32:28.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:32:28.543 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:28.544 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:28.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:29.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:29.729+0000 7f478dfc07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:32:29.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:29.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:29.964 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:32:29.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:32:29.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:29.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:30.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:31.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:31.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:31.457 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:32:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:32:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:31.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:31.886 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:32.888 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:32.888 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:32:32.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:32:32.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:32.889 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:34.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:34.457 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:34.458 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:32:34.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:32:34.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:34.458 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:34.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:35.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:35.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:35.923 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:32:35.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:32:35.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:35.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:36.486 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:37.488 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:37.489 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:32:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:32:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:37.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:37.921 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:38.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:38.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:38.923 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:32:38.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:32:38.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:38.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:39.513 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:40.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:40.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:40.516 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:32:40.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:32:40.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:40.516 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:40.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:41.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:41.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:41.958 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:32:41.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:32:41.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:41.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:42.512 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:43.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:43.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:43.514 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:32:43.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:32:43.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:43.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:43.956 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:44.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:44.957 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:44.958 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:32:44.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:32:44.958 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:44.959 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:45.689 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:46.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:46.691 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:46.691 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:32:46.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:32:46.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:46.692 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:47.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:48.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:48.357 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:48.357 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:32:48.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:32:48.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:48.358 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:49.705 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:50.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:50.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:50.707 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:32:50.707 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:32:50.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:50.708 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:51.804 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:52.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:52.806 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:52.806 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:32:52.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:32:52.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:52.807 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:53.287 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/892537467,v1:127.0.0.1:6803/892537467] [v2:127.0.0.1:6804/892537467,v1:127.0.0.1:6805/892537467] exists,up 36036f22-a3a4-4175-b55c-7114aebc4780 2024-04-25T09:32:53.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:32:53.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:32:53.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:32:53.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:32:53.289 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:32:53.290 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:32:53.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:32:53.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:32:53.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:32:53.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:53.291 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:32:53.291 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:32:53.292 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:32:53.293 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:32:53.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:32:53.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e94ae110-03e4-4783-aeb0-ccfdfb37a7df 2024-04-25T09:32:53.294 INFO:tasks.workunit.client.0.smithi031.stdout:add osd1 e94ae110-03e4-4783-aeb0-ccfdfb37a7df 2024-04-25T09:32:53.294 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e94ae110-03e4-4783-aeb0-ccfdfb37a7df' 2024-04-25T09:32:53.294 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:32:53.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFIypm3LXVExAAC0guS6FBfUls/33ujUYNCg== 2024-04-25T09:32:53.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFIypm3LXVExAAC0guS6FBfUls/33ujUYNCg=="}' 2024-04-25T09:32:53.316 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e94ae110-03e4-4783-aeb0-ccfdfb37a7df -i td/repro-long-log/1/new.json 2024-04-25T09:32:53.870 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:32:53.891 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:32:53.893 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFIypm3LXVExAAC0guS6FBfUls/33ujUYNCg== --osd-uuid e94ae110-03e4-4783-aeb0-ccfdfb37a7df 2024-04-25T09:32:53.920 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:53.933+0000 7fa8615d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:53.925 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:53.937+0000 7fa8615d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:53.926 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:53.941+0000 7fa8615d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:53.926 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:53.941+0000 7fa8615d87c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:32:53.927 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:53.941+0000 7fa8615d87c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:32:53.927 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:53.941+0000 7fa8615d87c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:32:54.017 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:32:54.029+0000 7fa8615d87c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:33:00.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:33:00.139 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:33:00.140 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd1 key to auth repository 2024-04-25T09:33:00.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:33:00.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:33:00.717 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.1 2024-04-25T09:33:00.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:33:00.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:33:00.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:33:00.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:33:00.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:33:00.748 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:00.761+0000 7f465a8277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:00.756 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:00.769+0000 7f465a8277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:00.758 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:00.773+0000 7f465a8277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:01.153 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:33:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:33:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:33:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:33:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:33:01.154 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:01.154 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:33:01.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:33:01.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:01.155 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:01.323 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:01.337+0000 7f465a8277c0 -1 Falling back to public interface 2024-04-25T09:33:01.618 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:02.619 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:33:02.620 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:33:02.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:02.620 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:03.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:03.829 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:03.841+0000 7f465a8277c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:33:04.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:04.049 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:04.050 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:33:04.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:33:04.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:04.050 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:04.740 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:05.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:05.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:05.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:33:05.742 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:33:05.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:05.743 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:06.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:07.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:07.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:07.177 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:33:07.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:33:07.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:07.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:07.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:08.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:08.719 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:08.720 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:33:08.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:33:08.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:08.720 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:09.265 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:10.267 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:10.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:10.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:33:10.268 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:33:10.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:10.268 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:10.731 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:11.732 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:11.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:11.733 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:33:11.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:33:11.733 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:11.734 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:12.205 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:13.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:13.207 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:13.207 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:33:13.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:33:13.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:13.208 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:13.715 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:14.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:14.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:14.717 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:33:14.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:33:14.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:14.718 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:15.172 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:16.174 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:16.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:16.175 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:33:16.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:33:16.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:16.176 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:17.175 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:18.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:18.177 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:18.178 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:33:18.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:33:18.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:18.178 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:19.593 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:20.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:20.595 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:20.595 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:33:20.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:33:20.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:20.596 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:21.603 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:22.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:22.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:22.605 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:33:22.605 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:33:22.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:22.606 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:23.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:24.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:24.827 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:24.827 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:33:24.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:33:24.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:24.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:25.697 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:26.698 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:26.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:26.699 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:33:26.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:33:26.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:26.700 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:27.222 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:28.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:28.223 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:28.223 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:33:28.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:33:28.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:28.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:33:28.671 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4159565677,v1:127.0.0.1:6811/4159565677] [v2:127.0.0.1:6812/4159565677,v1:127.0.0.1:6813/4159565677] exists,up e94ae110-03e4-4783-aeb0-ccfdfb37a7df 2024-04-25T09:33:28.672 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:33:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:33:28.674 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:33:28.676 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:33:28.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc3a1267-bebe-4aa8-bf63-a19cf4885108 2024-04-25T09:33:28.678 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc3a1267-bebe-4aa8-bf63-a19cf4885108' 2024-04-25T09:33:28.678 INFO:tasks.workunit.client.0.smithi031.stdout:add osd2 cc3a1267-bebe-4aa8-bf63-a19cf4885108 2024-04-25T09:33:28.678 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:33:28.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoIypmMFS4KhAAAzevtNnDFcdVWNbEZz3R4w== 2024-04-25T09:33:28.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoIypmMFS4KhAAAzevtNnDFcdVWNbEZz3R4w=="}' 2024-04-25T09:33:28.699 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc3a1267-bebe-4aa8-bf63-a19cf4885108 -i td/repro-long-log/2/new.json 2024-04-25T09:33:29.198 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:33:29.216 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:33:29.217 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoIypmMFS4KhAAAzevtNnDFcdVWNbEZz3R4w== --osd-uuid cc3a1267-bebe-4aa8-bf63-a19cf4885108 2024-04-25T09:33:29.250 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.264+0000 7f38bd8db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:29.254 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.268+0000 7f38bd8db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:29.255 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.268+0000 7f38bd8db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:29.256 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.268+0000 7f38bd8db7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:33:29.256 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.268+0000 7f38bd8db7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:33:29.256 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.268+0000 7f38bd8db7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:33:29.294 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:29.308+0000 7f38bd8db7c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:33:35.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:33:35.505 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:33:35.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:33:35.506 INFO:tasks.workunit.client.0.smithi031.stdout:adding osd2 key to auth repository 2024-04-25T09:33:35.506 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:33:36.083 INFO:tasks.workunit.client.0.smithi031.stdout:start osd.2 2024-04-25T09:33:36.083 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:33:36.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5179c5c7-309e-4d16-b406-120a75b604a8 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.486059/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:33:36.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:33:36.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:33:36.084 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:33:36.114 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:36.128+0000 7f5b57d5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:36.126 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:36.140+0000 7f5b57d5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:36.127 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:36.140+0000 7f5b57d5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:36.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:33:36.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:33:36.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:33:36.514 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:33:36.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:33:36.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:36.515 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:33:36.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:33:36.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:36.515 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:36.696 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:36.708+0000 7f5b57d5b7c0 -1 Falling back to public interface 2024-04-25T09:33:36.961 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:37.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:37.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:37.964 INFO:tasks.workunit.client.0.smithi031.stdout:1 2024-04-25T09:33:37.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:33:37.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:37.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:38.406 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:39.183 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25T09:33:39.196+0000 7f5b57d5b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:33:39.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:39.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:39.408 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:33:39.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:33:39.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:39.408 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:39.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:40.978 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:40.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:40.979 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:33:40.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:33:40.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:40.979 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:41.410 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:42.411 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:42.412 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:42.412 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:33:42.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:33:42.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:42.413 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:42.902 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:43.903 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:43.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:43.904 INFO:tasks.workunit.client.0.smithi031.stdout:5 2024-04-25T09:33:43.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:33:43.904 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:43.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:44.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:45.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:45.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:45.324 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:33:45.325 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:33:45.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:45.325 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:45.798 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:46.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:46.799 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:46.799 INFO:tasks.workunit.client.0.smithi031.stdout:7 2024-04-25T09:33:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:33:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:46.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:47.281 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:48.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:48.283 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:48.283 INFO:tasks.workunit.client.0.smithi031.stdout:8 2024-04-25T09:33:48.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:33:48.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:48.284 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:48.777 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:49.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:49.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:49.779 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:33:49.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:33:49.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:49.780 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:50.320 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:51.321 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:51.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:51.322 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:33:51.322 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:33:51.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:51.323 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:51.763 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:52.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:52.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:52.765 INFO:tasks.workunit.client.0.smithi031.stdout:11 2024-04-25T09:33:52.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:33:52.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:52.766 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:53.431 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:54.432 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:54.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:54.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:33:54.433 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:33:54.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:54.433 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:55.654 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:56.656 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:56.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:56.657 INFO:tasks.workunit.client.0.smithi031.stdout:13 2024-04-25T09:33:56.657 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:33:56.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:56.658 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:57.439 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:58.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:58.442 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:58.442 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:33:58.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:33:58.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:58.443 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:59.098 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:00.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:00.100 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:00.100 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:34:00.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:34:00.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:00.101 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:01.019 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:02.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:02.020 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:02.021 INFO:tasks.workunit.client.0.smithi031.stdout:16 2024-04-25T09:34:02.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:34:02.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:02.021 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:02.488 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1601143635,v1:127.0.0.1:6819/1601143635] [v2:127.0.0.1:6820/1601143635,v1:127.0.0.1:6821/1601143635] exists,up cc3a1267-bebe-4aa8-bf63-a19cf4885108 2024-04-25T09:34:02.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:34:02.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:34:02.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:34:02.489 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:34:03.011 INFO:tasks.workunit.client.0.smithi031.stderr:pool 'test' already exists 2024-04-25T09:34:03.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:34:03.030 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:34:03.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:34:03.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:34:03.574 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:34:03.714 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:03.747 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:03.784 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:03.801 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:34:03.992 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:04.022 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:04.052 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:04.067 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:34:04.267 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:04.294 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:04.323 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:04.337 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:34:04.503 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:04.532 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:04.561 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:04.576 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:34:04.748 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:04.781 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:04.812 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:04.828 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:34:04.829 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:34:04.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:04.830 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.068 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.211 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.445 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.553 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.554 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.779 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:05.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:05.905 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.048 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.156 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.157 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.290 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.291 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.407 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.508 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.742 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.852 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.853 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:06.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:06.998 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:07.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:07.136 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:07.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:07.236 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:07.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:07.361 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:34:07.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:34:07.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:07.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:34:07.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:34:07.463 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:07.611 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836500 2024-04-25T09:34:07.635 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672979 2024-04-25T09:34:07.659 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509446 2024-04-25T09:34:07.675 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:10.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:10.677 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:10.847 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:10.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:10.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:10.848 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:34:11.003 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:11.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:11.004 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:11.170 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:34:11.170 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:11.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:11.171 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:34:11.326 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:34:11.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:34:11.432 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:34:11.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:11.434 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:11.528 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:11.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:11.531 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:11.637 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:11.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:11.641 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:11.762 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:11.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:11.765 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:11.863 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:11.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:11.866 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:11.988 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:11.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:11.992 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.089 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.093 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.214 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.218 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.323 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.327 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.432 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.435 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.608 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.612 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.726 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.730 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.851 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.855 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:12.960 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:12.963 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:12.964 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.061 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:13.064 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.177 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:13.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.311 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:13.328 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.420 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:13.423 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.537 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:13.541 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.662 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:34:13.667 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:34:13.788 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:34:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:34:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:34:13.793 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:13.943 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836504 2024-04-25T09:34:13.968 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672982 2024-04-25T09:34:13.993 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509449 2024-04-25T09:34:14.008 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:17.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:17.011 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:17.181 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:34:17.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:17.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:17.181 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:34:17.344 INFO:tasks.workunit.client.0.smithi031.stdout:21 2024-04-25T09:34:17.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:17.345 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:17.506 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:34:17.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:17.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:17.507 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:34:17.669 INFO:tasks.workunit.client.0.smithi031.stdout:0 2024-04-25T09:34:17.669 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:34:17.808 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:17.837 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:17.865 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:17.880 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:34:18.028 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:18.057 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:18.087 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:18.102 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:34:18.268 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:18.298 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:18.327 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:34:18.489 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:18.518 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:18.553 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:18.567 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:34:18.707 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: {} 2024-04-25T09:34:18.737 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: {} 2024-04-25T09:34:18.767 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: {} 2024-04-25T09:34:18.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:19.002 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:19.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-04-25T09:34:19.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:19.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:34:19.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-04-25T09:34:19.007 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:19.158 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836507 2024-04-25T09:34:19.181 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672985 2024-04-25T09:34:19.205 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509452 2024-04-25T09:34:19.221 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:22.224 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:22.225 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:22.393 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:34:22.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:22.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:22.394 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:34:22.556 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:34:22.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:22.557 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:22.716 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:34:22.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:22.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:22.717 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-04-25T09:34:22.862 INFO:tasks.workunit.client.0.smithi031.stdout:2 2024-04-25T09:34:22.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:22.996 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:23.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-04-25T09:34:23.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:23.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:34:23.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-04-25T09:34:23.000 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:23.145 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836509 2024-04-25T09:34:23.169 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672988 2024-04-25T09:34:23.195 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509455 2024-04-25T09:34:23.210 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:26.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:26.212 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:26.382 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:34:26.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:26.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:26.383 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:34:26.549 INFO:tasks.workunit.client.0.smithi031.stdout:15 2024-04-25T09:34:26.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:26.550 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:26.713 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:34:26.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:26.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:26.714 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-04-25T09:34:26.862 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:34:26.863 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:26.973 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:26.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-04-25T09:34:26.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:26.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:34:26.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-04-25T09:34:26.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:27.121 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836512 2024-04-25T09:34:27.144 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672990 2024-04-25T09:34:27.168 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509457 2024-04-25T09:34:27.182 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:30.184 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:30.185 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:30.354 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:34:30.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:30.354 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:30.355 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:34:30.509 INFO:tasks.workunit.client.0.smithi031.stdout:12 2024-04-25T09:34:30.509 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:30.510 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:30.662 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:34:30.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:30.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:30.663 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-04-25T09:34:30.821 INFO:tasks.workunit.client.0.smithi031.stdout:10 2024-04-25T09:34:30.822 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:30.951 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-04-25T09:34:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:34:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-04-25T09:34:30.955 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:31.101 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836515 2024-04-25T09:34:31.127 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672993 2024-04-25T09:34:31.150 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509460 2024-04-25T09:34:31.165 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:34.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:34.168 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:34.338 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:34:34.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:34.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:34.339 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:34:34.493 INFO:tasks.workunit.client.0.smithi031.stdout:9 2024-04-25T09:34:34.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:34.494 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:34.661 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:34:34.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:34.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:34.662 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-04-25T09:34:34.811 INFO:tasks.workunit.client.0.smithi031.stdout:14 2024-04-25T09:34:34.811 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:34.920 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:34.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-04-25T09:34:34.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:34.923 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:34:34.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:34:34.924 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:35.069 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836518 2024-04-25T09:34:35.093 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672996 2024-04-25T09:34:35.117 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509463 2024-04-25T09:34:35.132 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:38.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:38.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:38.306 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:34:38.306 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:38.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:34:38.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:38.469 INFO:tasks.workunit.client.0.smithi031.stdout:6 2024-04-25T09:34:38.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:38.470 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:38.679 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:34:38.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:38.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:38.680 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:34:38.843 INFO:tasks.workunit.client.0.smithi031.stdout:18 2024-04-25T09:34:38.843 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:38.973 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:38.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:34:38.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:38.976 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:34:38.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:34:38.977 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:39.125 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836521 2024-04-25T09:34:39.151 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949672999 2024-04-25T09:34:39.177 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509466 2024-04-25T09:34:39.193 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:42.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:42.196 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:42.363 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:34:42.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:42.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:42.364 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:34:42.521 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:34:42.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:42.522 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:42.678 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:42.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:42.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:42.679 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:34:42.838 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:42.925 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:42.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:34:42.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:42.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:34:42.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:34:42.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:43.078 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836523 2024-04-25T09:34:43.102 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673002 2024-04-25T09:34:43.126 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509469 2024-04-25T09:34:43.140 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:46.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:46.143 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:46.313 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:34:46.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:46.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:46.314 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:34:46.473 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:34:46.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:46.474 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:46.652 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:46.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:46.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:46.653 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:34:46.809 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:46.809 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:46.944 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:46.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:34:46.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:46.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:34:46.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:34:46.947 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:47.095 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836526 2024-04-25T09:34:47.119 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673004 2024-04-25T09:34:47.144 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509471 2024-04-25T09:34:47.159 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:50.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:50.162 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:50.333 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:34:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:50.334 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:34:50.497 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:34:50.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:50.498 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:50.664 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:50.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:50.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:50.664 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:34:50.825 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:50.825 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:50.921 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:50.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:34:50.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:50.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:34:50.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:34:50.925 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:51.070 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836529 2024-04-25T09:34:51.094 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673007 2024-04-25T09:34:51.118 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509474 2024-04-25T09:34:51.133 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:54.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:54.135 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:54.307 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:34:54.307 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:54.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:54.308 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:34:54.464 INFO:tasks.workunit.client.0.smithi031.stdout:4 2024-04-25T09:34:54.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:54.465 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:54.625 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:54.625 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:54.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:54.626 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:34:54.781 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:54.782 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:34:54.924 INFO:tasks.workunit.client.0.smithi031.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:34:54.928 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:34:54.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:34:54.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:34:54.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:34:54.929 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:34:55.074 INFO:tasks.workunit.client.0.smithi031.stdout:osd.0: 21474836532 2024-04-25T09:34:55.098 INFO:tasks.workunit.client.0.smithi031.stdout:osd.1: 42949673010 2024-04-25T09:34:55.123 INFO:tasks.workunit.client.0.smithi031.stdout:osd.2: 64424509477 2024-04-25T09:34:55.138 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:34:58.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:58.141 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:58.310 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:34:58.310 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:58.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:34:58.311 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:34:58.468 INFO:tasks.workunit.client.0.smithi031.stdout:3 2024-04-25T09:34:58.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:58.469 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:58.633 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:58.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:34:58.633 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:34:58.634 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:34:58.799 INFO:tasks.workunit.client.0.smithi031.stdout:20 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:34:58.800 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:34:58.910 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:34:58.910 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:34:58.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:34:58.914 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:34:58.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:34:58.914 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:34:58.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:34:58.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:58.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:34:58.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:34:58.915 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:58.915 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:34:58.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:34:58.916 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:34:58.933 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:34:58.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:58.934 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:34:58.934 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:34:58.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:34:58.935 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repro-long-log 0 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:34:58.936 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:34:58.937 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:34:58.938 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:34:58.939 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:34:58.939 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:34:58.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:34:58.940 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:34:58.940 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:34:58.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:58.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:34:58.941 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:34:58.942 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:58.943 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:34:58.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:34:58.944 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:34:58.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:34:58.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:58.945 INFO:tasks.workunit.client.0.smithi031.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.486059 2024-04-25T09:34:58.945 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.486059 2024-04-25T09:34:58.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:34:58.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:34:58.946 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:34:58.948 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:34:58.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:34:58.958 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-04-25T09:34:58.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T09:34:59.313 DEBUG:teuthology.parallel:result is None 2024-04-25T09:34:59.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:34:59.327 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:34:59.327 DEBUG:teuthology.orchestra.run.smithi031:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:34:59.384 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:34:59.384 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T09:34:59.396 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T09:34:59.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T09:34:59.464 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T09:34:59.473 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T09:34:59.473 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:34:59.497 INFO:teuthology.orchestra.run.smithi031.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T09:34:59.497 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================== 2024-04-25T09:34:59.498 INFO:teuthology.orchestra.run.smithi031.stdout:+hv01.front.sepi 44.4.53.2 2 u 201 512 377 0.184 -2.652 0.593 2024-04-25T09:34:59.498 INFO:teuthology.orchestra.run.smithi031.stdout:+hv02.front.sepi 63.231.80.2 3 u 278 512 377 0.236 -0.075 0.499 2024-04-25T09:34:59.498 INFO:teuthology.orchestra.run.smithi031.stdout:*hv03.front.sepi 74.6.168.72 3 u 27 512 377 0.183 -1.421 0.948 2024-04-25T09:34:59.498 INFO:teuthology.orchestra.run.smithi031.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T09:34:59.498 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T09:34:59.507 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T09:34:59.508 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T09:34:59.517 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T09:34:59.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T09:34:59.557 INFO:teuthology.task.internal:Duration was 9676.127655 seconds 2024-04-25T09:34:59.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T09:34:59.567 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T09:34:59.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T09:34:59.610 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T09:34:59.610 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T09:34:59.611 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T09:34:59.662 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T09:34:59.662 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T09:34:59.741 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T09:34:59.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T09:34:59.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T09:34:59.854 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T09:34:59.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T09:34:59.866 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T09:34:59.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:34:59.919 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-04-25T09:34:59.934 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:34:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:34:59.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T09:34:59.992 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T09:34:59.994 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672918/remote/smithi031 2024-04-25T09:34:59.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T09:35:00.049 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T09:35:00.049 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T09:35:00.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T09:35:00.102 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T09:35:00.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T09:35:00.112 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T09:35:00.113 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T09:35:00.140 INFO:teuthology.orchestra.run.smithi031.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 09:35 /home/ubuntu/cephtest 2024-04-25T09:35:00.141 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T09:35:00.151 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T09:35:00.175 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 9676.12765455246 flavor: default owner: scheduled_nmordech@teuthology success: true 2024-04-25T09:35:00.175 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T09:35:00.257 INFO:teuthology.run:pass